Nov 01 05:00:21.108532 ip-10-0-141-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 05:00:21.108545 ip-10-0-141-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 05:00:21.108554 ip-10-0-141-97 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 05:00:21.108897 ip-10-0-141-97 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 05:00:31.305650 ip-10-0-141-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 05:00:31.305671 ip-10-0-141-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 768d42b3519347e8bf6e4180eef39daf -- Nov 01 05:02:24.604581 ip-10-0-141-97 systemd[1]: Starting Kubernetes Kubelet... Nov 01 05:02:25.003172 ip-10-0-141-97 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:02:25.003172 ip-10-0-141-97 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 05:02:25.003172 ip-10-0-141-97 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:02:25.003172 ip-10-0-141-97 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.007896 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.007986 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.007990 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.007994 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.007997 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.007990 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008001 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008004 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008007 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008010 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008012 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008015 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008017 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008020 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008022 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008025 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008028 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008030 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008032 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008035 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008037 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008040 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008042 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008045 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008047 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.008213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008050 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008052 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008055 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008059 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008064 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008067 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008071 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008074 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008077 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008080 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008083 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008085 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008088 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008091 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008094 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008096 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008099 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008102 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008104 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.008678 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008107 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008110 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008112 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008115 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008117 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008120 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008122 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008125 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008127 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008130 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008133 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008135 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008138 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008142 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008145 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008148 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008150 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008153 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008155 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008158 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.009126 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008161 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008164 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008166 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008168 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008171 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008174 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008176 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008179 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008182 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008185 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008188 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008192 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008196 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008200 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008202 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008205 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008208 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008211 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008214 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.009609 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008217 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008224 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008227 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008230 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008232 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008235 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008237 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008239 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008242 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008245 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008247 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008268 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008271 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008273 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008276 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008278 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008281 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008284 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.010050 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008286 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008289 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008292 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008295 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.008297 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009879 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009890 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009894 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009897 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009900 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009902 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009906 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009908 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009911 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009913 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009916 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009919 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009922 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009924 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.010537 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009927 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009930 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009933 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009935 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009938 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009941 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009943 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009946 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009948 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009951 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009953 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009956 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009958 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009961 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009965 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009968 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009970 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009973 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009976 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009979 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.010992 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009983 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009986 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009989 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009992 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009995 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.009997 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010000 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010002 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010005 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010009 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010012 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010014 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010017 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010019 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010022 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010025 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010027 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010030 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010033 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010035 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.011602 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010037 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010040 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010042 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010045 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010047 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010050 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010053 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010056 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010059 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010062 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010065 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010067 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010070 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010072 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010075 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010077 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010080 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010082 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010084 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.012067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010087 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010090 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010092 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010095 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010098 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010100 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010103 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010105 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010107 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010110 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010112 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010115 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010117 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010120 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010122 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010125 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010127 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010130 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010132 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010135 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.013366 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010138 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010140 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010143 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010146 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010148 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010151 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010153 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010156 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010158 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010161 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010163 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010166 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010169 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010726 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010733 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010737 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010741 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010745 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010748 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010751 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.013839 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010753 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010756 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010758 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010761 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010763 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010766 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010768 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010771 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010773 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010776 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010778 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010782 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010785 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010787 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010790 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010792 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010795 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010797 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010800 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.014335 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010802 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010805 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010807 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010811 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010815 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010818 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010821 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010824 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010826 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010829 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010832 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010834 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010837 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010839 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010842 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010844 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010847 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010849 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010852 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.014771 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010854 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010857 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010859 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010861 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010864 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010868 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010870 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010873 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010875 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010878 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010881 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010883 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010886 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010902 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010905 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010908 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010911 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010913 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010916 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.015234 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010918 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010921 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010923 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010926 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010929 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010932 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010934 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010938 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010940 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010943 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010945 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010948 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010950 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010954 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010958 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010961 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010963 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010966 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010968 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010971 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.015742 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010974 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010976 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010978 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010981 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010984 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010986 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010989 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010991 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010994 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010997 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.010999 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011003 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011006 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011008 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011011 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011013 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011016 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011018 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011021 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.016213 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011023 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011026 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011028 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011116 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011123 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011129 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011134 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011140 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011144 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011151 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011157 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011162 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011167 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011171 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011174 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011177 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011180 2581 flags.go:64] FLAG: --cgroup-root="" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011183 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011186 2581 flags.go:64] FLAG: --client-ca-file="" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011189 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011192 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011197 2581 flags.go:64] FLAG: --cluster-domain="" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011209 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 05:02:25.016698 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011213 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011216 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011219 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011223 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011226 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011229 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011232 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011235 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011238 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011242 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011245 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011248 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011270 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011273 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011276 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011279 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011282 2581 flags.go:64] FLAG: --enable-server="true" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011285 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011289 2581 flags.go:64] FLAG: --event-burst="100" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011296 2581 flags.go:64] FLAG: --event-qps="50" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011299 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011302 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011305 2581 flags.go:64] FLAG: --eviction-hard="" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011309 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 05:02:25.017228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011311 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011314 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011317 2581 flags.go:64] FLAG: --eviction-soft="" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011320 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011323 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011326 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011329 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011332 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011334 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011337 2581 flags.go:64] FLAG: --feature-gates="" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011342 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011345 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011348 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011351 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011353 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011356 2581 flags.go:64] FLAG: --help="false" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011359 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011363 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011366 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011369 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011372 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011375 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011378 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 05:02:25.017829 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011381 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011383 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011386 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011389 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011393 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011396 2581 flags.go:64] FLAG: --kube-reserved="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011399 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011402 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011405 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011408 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011411 2581 flags.go:64] FLAG: --lock-file="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011413 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011416 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011419 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011425 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011428 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011431 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011434 2581 flags.go:64] FLAG: --logging-format="text" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011437 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011440 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011443 2581 flags.go:64] FLAG: --manifest-url="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011446 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011450 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011453 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011457 2581 flags.go:64] FLAG: --max-pods="110" Nov 01 05:02:25.018384 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011460 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011463 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011466 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011469 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011472 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011475 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011478 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011486 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011489 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011494 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011497 2581 flags.go:64] FLAG: --pod-cidr="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011502 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011504 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011507 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011510 2581 flags.go:64] FLAG: --port="10250" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011513 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011516 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02e3120d51be079ee" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011519 2581 flags.go:64] FLAG: --qos-reserved="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011522 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011525 2581 flags.go:64] FLAG: --register-node="true" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011528 2581 flags.go:64] FLAG: --register-with-taints="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011532 2581 flags.go:64] FLAG: --registry-burst="10" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011535 2581 flags.go:64] FLAG: --registry-qps="5" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011538 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011541 2581 flags.go:64] FLAG: --reserved-memory="" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011544 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 05:02:25.018973 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011548 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011551 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011553 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011556 2581 flags.go:64] FLAG: --runonce="false" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011559 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011562 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011565 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011568 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011571 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011574 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011578 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011581 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011584 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011586 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011589 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011595 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011598 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011606 2581 flags.go:64] FLAG: --system-cgroups="" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011609 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011614 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011617 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011620 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011628 2581 flags.go:64] FLAG: --tls-min-version="" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011631 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011633 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 05:02:25.019664 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011636 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011639 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011642 2581 flags.go:64] FLAG: --v="2" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011647 2581 flags.go:64] FLAG: --version="false" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011651 2581 flags.go:64] FLAG: --vmodule="" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011655 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.011658 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011805 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011809 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011812 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011814 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011817 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011820 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011822 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011825 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011827 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011830 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011833 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011835 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011838 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011840 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.020263 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011843 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011847 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011850 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011854 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011856 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011859 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011863 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011867 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011870 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011873 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011876 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011878 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011882 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011884 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011887 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011889 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011892 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011894 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011897 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.020753 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011899 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011902 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011904 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011907 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011909 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011911 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011914 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011916 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011919 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011922 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011925 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011927 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011930 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011932 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011936 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011939 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011942 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011945 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011947 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.021210 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011950 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011953 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011955 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011958 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011960 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011963 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011965 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011968 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011970 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011974 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011978 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011980 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011983 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011986 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011988 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011991 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011993 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011995 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.011998 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.021677 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012000 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012003 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012005 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012008 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012010 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012013 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012016 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012018 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012022 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012024 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012028 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012030 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012033 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012035 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012038 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012042 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012044 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012047 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012049 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012052 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.022132 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012055 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012057 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012060 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012062 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012065 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012067 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012070 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012073 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012075 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012078 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012080 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012083 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012085 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012088 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012090 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.012117 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:02:25.022622 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012159 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012164 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012168 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012171 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012173 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012177 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012180 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012183 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012186 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012188 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012191 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012193 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012196 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012198 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012201 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012204 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012206 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012209 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012211 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.023000 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012214 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012216 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012219 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012221 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012223 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012226 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012229 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012231 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012234 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012236 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012238 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012241 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012243 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012245 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012261 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012264 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012269 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012273 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012279 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.023471 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012283 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012286 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012289 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012291 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012294 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012297 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012299 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012302 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012305 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012307 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012310 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012312 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012315 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012317 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012319 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012322 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012325 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012327 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012330 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012332 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.023898 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012334 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012337 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012339 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012342 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012344 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012346 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012349 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012352 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012355 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012358 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012360 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012364 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012368 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012371 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012373 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012375 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012378 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012381 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012383 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012385 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.024382 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012388 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012390 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012393 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012396 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012398 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012401 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012403 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012406 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012409 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012411 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012414 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012416 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012419 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012421 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012423 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012426 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012428 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012431 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012433 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012436 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.024890 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012438 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012441 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012444 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012449 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012454 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012457 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012460 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.012462 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.019708 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.019723 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.020426 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.020432 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020488 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020495 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020500 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020503 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020506 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020510 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020512 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.025464 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020517 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020520 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020522 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020525 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020528 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020531 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020533 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020536 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020538 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020541 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020543 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020546 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020548 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020551 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020553 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020556 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020559 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020562 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020565 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020568 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.025910 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020570 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020573 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020576 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020579 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020582 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020584 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020587 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020589 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020592 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020594 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020597 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020600 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020602 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020605 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020607 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020610 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020612 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020615 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020617 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020620 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.026387 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020622 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020625 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020628 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020630 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020633 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020635 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020638 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020640 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020643 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020646 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020650 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020652 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020655 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020657 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020660 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020663 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020665 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020668 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020670 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.026858 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020673 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020675 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020677 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020680 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020682 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020685 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020687 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020689 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020692 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020696 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020699 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020702 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020704 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020707 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020709 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020712 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020714 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020717 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020719 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020721 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.027317 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020724 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020726 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020729 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020732 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020735 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020737 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020740 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020743 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020745 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020748 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020750 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020752 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020755 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020758 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020760 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020763 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020766 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020768 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020771 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.027774 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020773 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.020801 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020954 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020960 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020965 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020968 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020971 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020974 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020977 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020980 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020982 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020985 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020987 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020990 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020993 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.028281 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020995 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.020997 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021001 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021003 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021006 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021009 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021011 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021014 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021016 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021018 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021021 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021023 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021026 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021028 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021031 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021033 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021035 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021038 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021040 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021043 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.028917 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021046 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021048 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021050 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021053 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021055 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021058 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021060 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021062 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021065 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021067 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021070 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021072 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021075 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021077 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021079 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021082 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021085 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021087 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021090 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021092 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.029445 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021095 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021097 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021100 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021102 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021104 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021107 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021109 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021111 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021114 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021116 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021119 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021121 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021123 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021126 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021128 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021131 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021133 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021135 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021138 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.030039 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021141 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021143 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021146 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021148 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021151 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021153 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021156 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021158 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021161 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021164 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021167 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021170 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021172 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021175 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021177 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021179 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021182 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021184 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021187 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021189 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.030821 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021191 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021194 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021196 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021198 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021201 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021204 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021206 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021208 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021211 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021215 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021218 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021221 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021224 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021227 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.021262 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:02:25.031487 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021296 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021300 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021302 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021305 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021308 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021310 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021313 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021316 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021318 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021321 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021323 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021326 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021328 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021331 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021333 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021336 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021338 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021340 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021343 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:02:25.031891 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021345 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021348 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021350 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021353 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021355 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021358 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021360 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021363 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021365 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021368 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021370 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021373 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021375 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021378 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021380 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021382 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021385 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021387 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021390 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021394 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:02:25.032718 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021397 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021400 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021403 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021405 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021408 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021411 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021413 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021416 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021418 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021421 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021424 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021426 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021428 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021430 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021433 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021435 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021438 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021440 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021443 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021446 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:02:25.033399 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021448 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021450 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021454 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021457 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021460 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021462 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021465 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021469 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021473 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021477 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021482 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021486 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021489 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021491 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021494 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021497 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021500 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021502 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:02:25.033879 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021504 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021507 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021509 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021512 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021514 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021517 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021519 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021522 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021524 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021527 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021529 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021531 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021534 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021536 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021538 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021541 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021543 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021546 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021548 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:02:25.034368 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021551 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021553 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021558 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021561 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021563 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021566 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021570 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021572 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021575 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:25.021577 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.021629 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.023929 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.024901 2581 server.go:1014] "Starting client certificate rotation" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.024994 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.025024 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.026990 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:02:25.034813 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.028098 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:02:25.039030 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.039014 2581 log.go:25] "Validated CRI v1 runtime API" Nov 01 05:02:25.042889 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.042871 2581 log.go:25] "Validated CRI v1 image API" Nov 01 05:02:25.043837 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.043813 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 05:02:25.049272 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.049236 2581 fs.go:135] Filesystem UUIDs: map[0c22b2a2-90d5-4af3-9781-78f12c58ecb0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7e279d0-a07c-4159-8ea9-34babcdae9e9:/dev/nvme0n1p3] Nov 01 05:02:25.049393 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.049270 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 05:02:25.054601 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.054488 2581 manager.go:217] Machine: {Timestamp:2026-11-01 05:02:25.053009153 +0000 UTC m=+0.348224122 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111703 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29feef83cd25998d16a81b61e13824 SystemUUID:ec29feef-83cd-2599-8d16-a81b61e13824 BootID:768d42b3-5193-47e8-bf6e-4180eef39daf Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:17:1f:b0:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:17:1f:b0:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:50:7c:7a:30:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 05:02:25.054673 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.054607 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 05:02:25.054700 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.054692 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 05:02:25.058015 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.057990 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 05:02:25.058146 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.058017 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-97.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 05:02:25.058188 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.058148 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 05:02:25.058188 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.058157 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 05:02:25.058188 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.058182 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:02:25.058278 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.058196 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:02:25.059504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.059494 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 05:02:25.063790 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.063777 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 05:02:25.063905 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.063896 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 05:02:25.064722 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.064713 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 01 05:02:25.064763 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.064726 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 05:02:25.064763 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.064737 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 05:02:25.064763 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.064746 2581 kubelet.go:405] "Adding apiserver pod source" Nov 01 05:02:25.064763 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.064755 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 05:02:25.065080 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.065065 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:02:25.067516 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.067492 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 05:02:25.074921 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.074907 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:02:25.075854 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.075841 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 05:02:25.075887 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.075872 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 05:02:25.076048 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076036 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 05:02:25.076077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076054 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 05:02:25.076077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076060 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 05:02:25.076077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076066 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 05:02:25.076077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076071 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 05:02:25.076077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076077 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 05:02:25.076775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076766 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 05:02:25.076812 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076776 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 05:02:25.076812 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076790 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 05:02:25.076812 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076796 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 05:02:25.076812 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076809 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 05:02:25.076923 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.076817 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 05:02:25.077625 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.077614 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 05:02:25.077662 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.077628 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 05:02:25.081801 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.081774 2581 server.go:1263] "Started kubelet" Nov 01 05:02:25.081929 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.081821 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 05:02:25.082467 ip-10-0-141-97 systemd[1]: Started Kubernetes Kubelet. Nov 01 05:02:25.082975 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.081909 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 05:02:25.082975 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.082731 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 05:02:25.084448 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.084423 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 05:02:25.088031 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.088011 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 01 05:02:25.089712 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.089687 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 05:02:25.094714 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.094695 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 05:02:25.095277 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.095242 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 05:02:25.095361 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.095338 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:25.096072 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096052 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 05:02:25.096163 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096059 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 05:02:25.096163 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096092 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 05:02:25.096280 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096197 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 01 05:02:25.096280 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096205 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 01 05:02:25.096280 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096212 2581 factory.go:55] Registering systemd factory Nov 01 05:02:25.096280 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096273 2581 factory.go:223] Registration of the systemd container factory successfully Nov 01 05:02:25.096510 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096497 2581 factory.go:153] Registering CRI-O factory Nov 01 05:02:25.096570 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096512 2581 factory.go:223] Registration of the crio container factory successfully Nov 01 05:02:25.096570 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096557 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 05:02:25.096662 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096571 2581 factory.go:103] Registering Raw factory Nov 01 05:02:25.096662 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.096582 2581 manager.go:1199] Started watching for new ooms in manager Nov 01 05:02:25.097582 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.097568 2581 manager.go:319] Starting recovery of all containers Nov 01 05:02:25.106790 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.106775 2581 manager.go:324] Recovery completed Nov 01 05:02:25.110555 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.110541 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.112084 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112059 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.112166 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112097 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.112166 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112111 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.112598 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112586 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.112652 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112588 2581 cpu_manager.go:226] "Starting" policy="none" Nov 01 05:02:25.112652 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112611 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 05:02:25.112652 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.112628 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 05:02:25.113295 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.113281 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.113372 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.113307 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.113372 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.113324 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.114695 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.114682 2581 policy_none.go:50] "Start" Nov 01 05:02:25.114745 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.114698 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 05:02:25.114745 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.114708 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 05:02:25.116659 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.116647 2581 policy_none.go:44] "Start" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.159911 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.162343 2581 manager.go:341] "Starting Device Plugin manager" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.162988 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.162999 2581 server.go:87] "Starting device plugin registration server" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.163233 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.163243 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.163366 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.163455 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.163464 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.163969 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.164014 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.168009 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.169356 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.169373 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.169390 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 05:02:25.176447 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.169418 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 05:02:25.224212 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.224183 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-97.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 05:02:25.224377 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.224183 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-97.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 05:02:25.225324 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.224183 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-97.us-west-1.compute.internal.18e3d33f36ed7cb3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-97.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:25.081523379 +0000 UTC m=+0.376738350,LastTimestamp:2026-11-01 05:02:25.081523379 +0000 UTC m=+0.376738350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-97.us-west-1.compute.internal,}" Nov 01 05:02:25.264349 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.264275 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.265231 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.265212 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.265358 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.265243 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.265358 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.265270 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.265358 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.265301 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.269849 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.269828 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal"] Nov 01 05:02:25.269943 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.269881 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.270720 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.270699 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.270825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.270728 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.270825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.270741 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.272930 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.272915 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.273077 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273063 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.273117 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273093 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.273632 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273616 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.273692 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.273692 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273664 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.273774 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273755 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.273883 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273787 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.273883 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.273802 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.275711 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.275695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.275784 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.275729 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.277398 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.277383 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.277477 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.277414 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.277477 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.277437 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.291725 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.291700 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.297582 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.297561 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.298063 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.298045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.298129 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.298090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.298197 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.298180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.333857 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.333824 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 05:02:25.333857 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.333770 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-97.us-west-1.compute.internal.18e3d33f38bfc69d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-97.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-97.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:25.112082077 +0000 UTC m=+0.407297047,LastTimestamp:2026-11-01 05:02:25.112082077 +0000 UTC m=+0.407297047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-97.us-west-1.compute.internal,}" Nov 01 05:02:25.334059 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.333916 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-97.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 05:02:25.334059 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.334039 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 05:02:25.334161 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.334083 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 05:02:25.352707 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.352680 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 05:02:25.352832 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.352743 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k56c7" Nov 01 05:02:25.387226 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.387195 2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-97.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.387361 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.387174 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-97.us-west-1.compute.internal.18e3d33f38c01eec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-97.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-97.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:25.112104684 +0000 UTC m=+0.407319653,LastTimestamp:2026-11-01 05:02:25.112104684 +0000 UTC m=+0.407319653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-97.us-west-1.compute.internal,}" Nov 01 05:02:25.404743 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.404844 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.404844 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.404844 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404813 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.404844 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.404991 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.404870 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.527167 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.527087 2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-97.us-west-1.compute.internal\" not found" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.588007 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.587965 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:25.589023 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.589007 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:25.589118 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.589035 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:25.589118 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.589045 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:25.589118 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.589078 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.596175 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.596160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.603987 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.603971 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.645562 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:25.645536 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:25.645562 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.645562 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-97.us-west-1.compute.internal\": node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:25.760951 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.760918 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:25.861280 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.861188 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:25.962273 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:25.962208 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.024953 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.024923 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 05:02:26.025396 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.025140 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa20f316a090d4ba282b8462800e02e0-246993faf7c49e49.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-97.us-west-1.compute.internal.18e3d33f38bfc69d\": read tcp 10.0.141.97:33054->3.147.100.37:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-97.us-west-1.compute.internal.18e3d33f38bfc69d default 5055 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-97.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-97.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-97.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:25 +0000 UTC,LastTimestamp:2026-11-01 05:02:25.270713809 +0000 UTC m=+0.565928780,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-97.us-west-1.compute.internal,}" Nov 01 05:02:26.063239 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.063208 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.163997 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.163924 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.264168 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.264130 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.358351 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.358302 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 04:57:25 +0000 UTC" deadline="2028-04-20 15:50:44.468525636 +0000 UTC" Nov 01 05:02:26.358351 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.358335 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12874h48m18.110193197s" Nov 01 05:02:26.364949 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.364932 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.373089 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.373070 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:02:26.374092 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.374076 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:02:26.374187 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.374100 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:02:26.374187 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.374112 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:02:26.385234 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.385216 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:02:26.465298 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.465202 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.540913 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.540888 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:02:26.566031 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.565996 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.666520 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:26.666484 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:02:26.681067 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:26.681035 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a641920c8382c6805dc090befb3983.slice/crio-08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5 WatchSource:0}: Error finding container 08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5: Status 404 returned error can't find the container with id 08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5 Nov 01 05:02:26.681289 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:26.681266 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0733015cf0f4fb0a3c7bd42111c0f986.slice/crio-b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89 WatchSource:0}: Error finding container b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89: Status 404 returned error can't find the container with id b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89 Nov 01 05:02:26.687153 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.687136 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:02:26.762677 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.762606 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:02:26.796338 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.796305 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:26.861916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.861887 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:02:26.864024 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.864009 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:02:26.947340 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.947312 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:02:26.947522 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:26.947370 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:02:27.068493 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.068409 2581 apiserver.go:52] "Watching apiserver" Nov 01 05:02:27.096368 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.096335 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 05:02:27.174867 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.174813 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"0733015cf0f4fb0a3c7bd42111c0f986","Type":"ContainerStarted","Data":"b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89"} Nov 01 05:02:27.176019 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.175993 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerStarted","Data":"08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5"} Nov 01 05:02:27.178647 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.178629 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 05:02:27.178978 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.178958 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-5fx5x","openshift-network-operator/iptables-alerter-v9bmm","openshift-multus/multus-jz9g4","openshift-multus/network-metrics-daemon-tfwcx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc","openshift-multus/multus-additional-cni-plugins-rfqts","openshift-ovn-kubernetes/ovnkube-node-s7lrf","kube-system/global-pull-secret-syncer-kwpzs","kube-system/konnectivity-agent-mhnv9","kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal","openshift-dns/node-resolver-b7dxf","openshift-image-registry/node-ca-7b7g8","openshift-cluster-node-tuning-operator/tuned-s6ddw"] Nov 01 05:02:27.183063 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.183045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.185246 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.185228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.188164 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.188144 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.190662 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.190644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.193039 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.193022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.193138 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.193101 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:27.195723 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.195700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.197988 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.197973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.198068 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.198029 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:27.200347 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.200327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.202606 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.202587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:27.202679 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.202643 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:27.206788 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.206764 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.209266 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.209241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.211840 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.211822 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.214820 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.214924 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.214924 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.214924 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215072 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215072 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.214965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5btn\" (UniqueName: \"kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.215072 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8l8\" (UniqueName: \"kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215072 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255cm\" (UniqueName: \"kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.215235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215118 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkr95\" (UniqueName: \"kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215319 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxskc\" (UniqueName: \"kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.215447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215380 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.215447 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215542 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.215670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215677 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwfq9\" (UniqueName: \"kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.215986 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.215988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.216426 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.216217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.239566 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239548 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 05:02:27.239665 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239596 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.239728 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239680 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 05:02:27.239728 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239688 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 05:02:27.239872 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239857 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.239949 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.239902 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbhl7\"" Nov 01 05:02:27.241193 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.241176 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.241300 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.241183 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 05:02:27.241300 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.241280 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.241446 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.241327 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6ndpw\"" Nov 01 05:02:27.244003 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.243985 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.244154 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.244099 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.244154 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.244102 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gnn2d\"" Nov 01 05:02:27.244154 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.244147 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 05:02:27.246237 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.246222 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nkwq4\"" Nov 01 05:02:27.246315 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.246302 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 05:02:27.251365 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.251351 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.251518 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.251504 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 05:02:27.251619 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.251601 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9csn\"" Nov 01 05:02:27.251675 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.251632 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.256015 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.255999 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.256098 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.256077 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.256433 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.256417 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q6tm4\"" Nov 01 05:02:27.262740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.262721 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 05:02:27.262740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.262735 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tw68j\"" Nov 01 05:02:27.262875 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.262766 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 05:02:27.265482 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265465 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.265617 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265601 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 05:02:27.265695 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265623 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 05:02:27.265695 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265639 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 05:02:27.265695 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265649 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.265695 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265676 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vjvv\"" Nov 01 05:02:27.265834 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.265727 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 05:02:27.290247 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.290221 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 05:02:27.290409 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.290299 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q5vvv" Nov 01 05:02:27.297281 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.297262 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 05:02:27.317484 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.317572 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.317572 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.317572 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.317572 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwgj\" (UniqueName: \"kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.317749 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.317749 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.317749 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.317749 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.317749 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317740 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.317933 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.317887 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.318279 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.318392 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.318466 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5btn\" (UniqueName: \"kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.318562 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.318622 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.318622 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318597 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.318723 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.318723 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318705 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.318835 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkr95\" (UniqueName: \"kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.318835 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.318835 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.318835 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.319027 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319027 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319027 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319027 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.318998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxskc\" (UniqueName: \"kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9t5h\" (UniqueName: \"kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkts\" (UniqueName: \"kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.319504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319372 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.319504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.319504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfq9\" (UniqueName: \"kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.319504 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.319717 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.319717 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.319717 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.319717 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.319717 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.319946 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.319946 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.319946 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.319946 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.319946 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.320188 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.319978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.320188 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.320547 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.320614 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.320670 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.320726 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.320795 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.320795 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8l8\" (UniqueName: \"kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.320888 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255cm\" (UniqueName: \"kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.320940 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.320940 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.320915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.321033 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321018 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321108 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321089 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.321171 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.321229 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321292 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321387 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.321387 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321346 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321506 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321506 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.321612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.321612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.321612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.321612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321601 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.321612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321609 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321644 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321670 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321701 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321848 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.321916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321877 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321956 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.321962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322063 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.322098 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322102 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.322389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322369 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322396 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.322497 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322606 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322624 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.322740 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.322737 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:27.822683691 +0000 UTC m=+3.117898671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322848 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.322901 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.323020 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:27.822937042 +0000 UTC m=+3.118152006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323069 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323082 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323140 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:27.323492 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323546 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323584 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323628 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.323663 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323630 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.323706 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.323742 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.323728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.348389 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.348365 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 05:02:27.348461 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.348394 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 05:02:27.348525 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.348508 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fhbr8\"" Nov 01 05:02:27.381516 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.381489 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 05:02:27.385535 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.385514 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5btn\" (UniqueName: \"kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.386241 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.386217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxskc\" (UniqueName: \"kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.386588 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.386556 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255cm\" (UniqueName: \"kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.386680 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.386556 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfq9\" (UniqueName: \"kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.387055 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.387037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8l8\" (UniqueName: \"kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.387158 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.387140 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkr95\" (UniqueName: \"kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.425016 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.424986 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425016 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425155 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwgj\" (UniqueName: \"kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425155 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425155 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425292 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425144 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425292 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425154 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425292 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425347 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425305 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425421 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425543 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425588 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425614 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425633 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9t5h\" (UniqueName: \"kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkts\" (UniqueName: \"kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425778 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.425825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425945 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.425972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426034 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426089 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426091 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.426775 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426238 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426257 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426364 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426655 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426668 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426720 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426759 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427455 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426885 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.427980 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.427980 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426959 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.427980 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.426980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.428151 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.428133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.428233 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.428197 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.428266 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.428237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.429243 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.429218 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.488819 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.488787 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:27.488819 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.488811 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:27.488819 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.488822 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:27.489034 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.488888 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:27.988868195 +0000 UTC m=+3.284083177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:27.491142 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.491115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9t5h\" (UniqueName: \"kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.491232 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.491159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwgj\" (UniqueName: \"kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.491232 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.491188 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkts\" (UniqueName: \"kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.509437 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.509412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:02:27.511601 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.511582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:02:27.515171 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.515152 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:02:27.517633 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.517613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jz9g4" Nov 01 05:02:27.523396 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.523372 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f0ce12_35eb_4c8b_9cfc_aaba8f41ded4.slice/crio-4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490 WatchSource:0}: Error finding container 4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490: Status 404 returned error can't find the container with id 4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490 Nov 01 05:02:27.527400 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.527377 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:02:27.529730 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.529704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:02:27.530475 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.530455 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7016375e_62e0_481f_a376_91f747a787bb.slice/crio-850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a WatchSource:0}: Error finding container 850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a: Status 404 returned error can't find the container with id 850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a Nov 01 05:02:27.531807 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.531616 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03449c8_5c4c_44ae_87b6_d784ca1dec2e.slice/crio-b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb WatchSource:0}: Error finding container b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb: Status 404 returned error can't find the container with id b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb Nov 01 05:02:27.536616 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.536596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:27.539913 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.539893 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda614c70f_fd8c_4df3_9cf9_5c287da0e33f.slice/crio-f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992 WatchSource:0}: Error finding container f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992: Status 404 returned error can't find the container with id f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992 Nov 01 05:02:27.541165 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.541077 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed794e8_a779_46c6_8834_78145d194d1d.slice/crio-5a35d380edaa4fcc2e6b52ee02b53aeef50be5ffc1db8b84ccf01ca8dc1638d2 WatchSource:0}: Error finding container 5a35d380edaa4fcc2e6b52ee02b53aeef50be5ffc1db8b84ccf01ca8dc1638d2: Status 404 returned error can't find the container with id 5a35d380edaa4fcc2e6b52ee02b53aeef50be5ffc1db8b84ccf01ca8dc1638d2 Nov 01 05:02:27.547466 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.547447 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89f9f69_4ddd_4676_a02b_d836a0cf627e.slice/crio-132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda WatchSource:0}: Error finding container 132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda: Status 404 returned error can't find the container with id 132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda Nov 01 05:02:27.552026 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.552010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:02:27.558177 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.558158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:27.564096 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.564081 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385d84d8_4a6b_4975_b02c_a80699dd5bca.slice/crio-b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24 WatchSource:0}: Error finding container b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24: Status 404 returned error can't find the container with id b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24 Nov 01 05:02:27.570446 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:27.570389 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa48682c_8d53_4fb0_8815_618b9720f89a.slice/crio-5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a WatchSource:0}: Error finding container 5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a: Status 404 returned error can't find the container with id 5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a Nov 01 05:02:27.830099 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.829995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:27.830099 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:27.830063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:27.830354 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.830196 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:27.830354 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.830281 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:27.830354 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.830301 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:28.830278224 +0000 UTC m=+4.125493198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:27.830354 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:27.830335 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:28.83032315 +0000 UTC m=+4.125538127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:28.031354 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.031312 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:28.031584 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.031557 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:28.031704 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.031592 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:28.031704 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.031605 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:28.031704 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.031669 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:29.031647722 +0000 UTC m=+4.326862695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:28.179235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.179064 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jz9g4" event={"ID":"c03449c8-5c4c-44ae-87b6-d784ca1dec2e","Type":"ContainerStarted","Data":"b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb"} Nov 01 05:02:28.180484 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.180453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9bmm" event={"ID":"7016375e-62e0-481f-a376-91f747a787bb","Type":"ContainerStarted","Data":"850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a"} Nov 01 05:02:28.181849 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.181818 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerStarted","Data":"22a01d278c2804e1c097df26d6d22c8b79afed7fe75204eede81968f10ac7768"} Nov 01 05:02:28.183265 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.183239 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7b7g8" event={"ID":"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4","Type":"ContainerStarted","Data":"4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490"} Nov 01 05:02:28.184745 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.184723 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" event={"ID":"3ed794e8-a779-46c6-8834-78145d194d1d","Type":"ContainerStarted","Data":"5a35d380edaa4fcc2e6b52ee02b53aeef50be5ffc1db8b84ccf01ca8dc1638d2"} Nov 01 05:02:28.186211 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.186189 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a"} Nov 01 05:02:28.187729 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.187703 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b7dxf" event={"ID":"385d84d8-4a6b-4975-b02c-a80699dd5bca","Type":"ContainerStarted","Data":"b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24"} Nov 01 05:02:28.189514 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.189491 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhnv9" event={"ID":"d89f9f69-4ddd-4676-a02b-d836a0cf627e","Type":"ContainerStarted","Data":"132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda"} Nov 01 05:02:28.190874 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.190848 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" event={"ID":"a614c70f-fd8c-4df3-9cf9-5c287da0e33f","Type":"ContainerStarted","Data":"f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992"} Nov 01 05:02:28.293429 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.293385 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:27 +0000 UTC" deadline="2028-04-20 14:08:51.671971599 +0000 UTC" Nov 01 05:02:28.293429 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.293421 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12873h6m23.378555976s" Nov 01 05:02:28.839434 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.839384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:28.839621 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:28.839499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:28.839706 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.839647 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:28.839758 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.839716 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:30.839697291 +0000 UTC m=+6.134912269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:28.839823 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.839790 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:28.839873 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:28.839830 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:30.839817631 +0000 UTC m=+6.135032604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:29.041287 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.041114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:29.041287 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.041280 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:29.041532 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.041322 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:29.041532 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.041337 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:29.041532 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.041466 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:31.04143053 +0000 UTC m=+6.336645491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:29.174481 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.174396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:29.174640 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.174581 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:29.174709 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.174688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:29.174800 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.174775 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:29.174883 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.174852 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:29.174986 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:29.174931 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:29.294034 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.293940 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:27 +0000 UTC" deadline="2028-04-24 21:29:37.886905403 +0000 UTC" Nov 01 05:02:29.294034 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:29.293999 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12976h27m8.592910578s" Nov 01 05:02:30.855955 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:30.855698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:30.856429 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:30.855993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:30.856429 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:30.856121 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:30.856429 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:30.856187 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:34.856168362 +0000 UTC m=+10.151383323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:30.856429 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:30.856259 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:30.856429 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:30.856301 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:34.856286095 +0000 UTC m=+10.151501074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:31.057858 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.057811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:31.058101 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.058067 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:31.058187 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.058143 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:31.058187 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.058177 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:31.058293 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.058283 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:35.05826256 +0000 UTC m=+10.353477535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:31.172382 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.172352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:31.172539 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.172504 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:31.172604 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.172579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:31.172682 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.172660 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:31.172750 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.172725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:31.172885 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:31.172799 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:31.215341 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.215228 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerStarted","Data":"89ce9078ff3370d3db1623c9a162ba28190f87374b0e651b18e051d66054945f"} Nov 01 05:02:31.228266 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.227569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"0733015cf0f4fb0a3c7bd42111c0f986","Type":"ContainerStarted","Data":"119f6dd11b28c54caf89b3b1d03ac300437f7366a5d9aef30480ac7db418bac1"} Nov 01 05:02:31.248958 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.248146 2581 generic.go:356] "Generic (PLEG): container finished" podID="73a641920c8382c6805dc090befb3983" containerID="2a06c6e65303f6377b2e3cce9e30e418b75077ac055d7fa376351adcadabb999" exitCode=0 Nov 01 05:02:31.248958 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.248210 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerDied","Data":"2a06c6e65303f6377b2e3cce9e30e418b75077ac055d7fa376351adcadabb999"} Nov 01 05:02:31.474548 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:31.474485 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" podStartSLOduration=4.474464947 podStartE2EDuration="4.474464947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:31.350850517 +0000 UTC m=+6.646065515" watchObservedRunningTime="2026-11-01 05:02:31.474464947 +0000 UTC m=+6.769679932" Nov 01 05:02:32.256451 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:32.256416 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerStarted","Data":"2d874eae6bb71d544fa37cdae3de5511cdcd6c96c547b8187445885302d29bac"} Nov 01 05:02:32.264296 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:32.263922 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="89ce9078ff3370d3db1623c9a162ba28190f87374b0e651b18e051d66054945f" exitCode=0 Nov 01 05:02:32.264296 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:32.264003 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"89ce9078ff3370d3db1623c9a162ba28190f87374b0e651b18e051d66054945f"} Nov 01 05:02:32.375626 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:32.375549 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" podStartSLOduration=5.375528126 podStartE2EDuration="5.375528126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:32.374553247 +0000 UTC m=+7.669768228" watchObservedRunningTime="2026-11-01 05:02:32.375528126 +0000 UTC m=+7.670743105" Nov 01 05:02:33.169979 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:33.169939 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:33.170160 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:33.170010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:33.170160 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:33.170031 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:33.170160 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:33.170143 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:33.170321 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:33.170225 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:33.170371 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:33.170327 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:33.274574 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:33.274496 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhnv9" event={"ID":"d89f9f69-4ddd-4676-a02b-d836a0cf627e","Type":"ContainerStarted","Data":"7a903ee5c58959583874392f6c5326362659029e50e70bc36521493d00df3eb6"} Nov 01 05:02:33.396100 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:33.396006 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mhnv9" podStartSLOduration=2.533237584 podStartE2EDuration="7.395985214s" totalImagesPullingTime="4.86274763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:33.394651336 +0000 UTC m=+8.689866344" watchObservedRunningTime="2026-11-01 05:02:33.395985214 +0000 UTC m=+8.691200195" Nov 01 05:02:34.297176 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:34.297134 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" event={"ID":"a614c70f-fd8c-4df3-9cf9-5c287da0e33f","Type":"ContainerStarted","Data":"98903c337127b6d3c37d0410f0cb9a91eb23fabda8b069319260f00f8141256f"} Nov 01 05:02:34.300344 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:34.300312 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7b7g8" event={"ID":"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4","Type":"ContainerStarted","Data":"b16e1909c4b6a4edacbc8284839b669e59b9b62cba5c7e2c2dcca0afd3fd1d4c"} Nov 01 05:02:34.899536 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:34.899496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:34.899725 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:34.899562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:34.899725 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:34.899689 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:34.899834 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:34.899757 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:42.899735886 +0000 UTC m=+18.194950867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:34.899895 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:34.899838 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:34.899895 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:34.899871 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:42.899860948 +0000 UTC m=+18.195075906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:35.101062 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.101027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:35.101230 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.101184 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:35.101230 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.101200 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:35.101230 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.101210 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:35.101415 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.101282 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:43.10124766 +0000 UTC m=+18.396462617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:35.179228 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.179148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:35.179573 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.179547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:35.179714 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.179655 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:35.179793 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.179715 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:35.179793 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.179785 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:35.179889 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:35.179478 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:35.392727 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.392692 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:35.394943 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.394915 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:35.508353 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:35.508235 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7b7g8" podStartSLOduration=3.521533765 podStartE2EDuration="9.508215957s" totalImagesPullingTime="5.986682192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:34.414677796 +0000 UTC m=+9.709892777" watchObservedRunningTime="2026-11-01 05:02:35.508215957 +0000 UTC m=+10.803430937" Nov 01 05:02:36.380612 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:36.380474 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 05:02:37.169938 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.169860 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:37.169938 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.169897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:37.170428 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.169860 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:37.170428 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:37.170000 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:37.170428 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:37.170098 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:37.170428 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:37.170155 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:37.180393 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.180291 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:02:36.380493934Z","UUID":"5aba40d0-e07f-42d1-92af-c4f2bf1467ec","Handler":null,"Name":"","Endpoint":""} Nov 01 05:02:37.183478 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.183454 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 05:02:37.183623 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.183489 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 05:02:37.309271 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.309207 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="355342d9cbc9384a0f579b4d3adef9411a9caad0f5fb8b3d20bb4f79eef96164" exitCode=0 Nov 01 05:02:37.309436 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.309353 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"355342d9cbc9384a0f579b4d3adef9411a9caad0f5fb8b3d20bb4f79eef96164"} Nov 01 05:02:37.312530 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.311988 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b7dxf" event={"ID":"385d84d8-4a6b-4975-b02c-a80699dd5bca","Type":"ContainerStarted","Data":"cf03f3fb0fc0288e55e7ee664e91afe49337a5905b54982416fdbeec35269b5d"} Nov 01 05:02:37.317436 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.317391 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" event={"ID":"a614c70f-fd8c-4df3-9cf9-5c287da0e33f","Type":"ContainerStarted","Data":"3afb0a734bd330cf27e029a1525cd88e59491ff935e9a6a0af17783ffb14979a"} Nov 01 05:02:37.319425 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.319391 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jz9g4" event={"ID":"c03449c8-5c4c-44ae-87b6-d784ca1dec2e","Type":"ContainerStarted","Data":"50ce1bce95d6a29bfc523fdea1eafdbf38441d7e06ae84602b84cd81b191c18a"} Nov 01 05:02:37.555809 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:37.555624 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jz9g4" podStartSLOduration=2.849838023 podStartE2EDuration="11.555604847s" totalImagesPullingTime="8.705766824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:37.555306715 +0000 UTC m=+12.850521696" watchObservedRunningTime="2026-11-01 05:02:37.555604847 +0000 UTC m=+12.850819826" Nov 01 05:02:38.324485 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:38.324214 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" event={"ID":"a614c70f-fd8c-4df3-9cf9-5c287da0e33f","Type":"ContainerStarted","Data":"1d2c054ca1c846787117f4065dd0a3697815e8590f4a8de57d4d2cc09ce7b370"} Nov 01 05:02:38.326085 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:38.326053 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9bmm" event={"ID":"7016375e-62e0-481f-a376-91f747a787bb","Type":"ContainerStarted","Data":"ff8dd2ba33e07c4418c9bae5b681830d75219cd5bc6d793c901bd71e40635fe3"} Nov 01 05:02:38.442041 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:38.441989 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b7dxf" podStartSLOduration=3.751987654 podStartE2EDuration="12.44197096s" totalImagesPullingTime="8.689983306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:37.669584892 +0000 UTC m=+12.964799882" watchObservedRunningTime="2026-11-01 05:02:38.44197096 +0000 UTC m=+13.737185942" Nov 01 05:02:38.442575 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:38.442539 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" podStartSLOduration=2.387238986 podStartE2EDuration="12.442527577s" totalImagesPullingTime="10.055288591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:38.441840514 +0000 UTC m=+13.737055496" watchObservedRunningTime="2026-11-01 05:02:38.442527577 +0000 UTC m=+13.737742558" Nov 01 05:02:38.554905 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:38.554839 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v9bmm" podStartSLOduration=3.6943672579999998 podStartE2EDuration="12.554816454s" totalImagesPullingTime="8.860449196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:38.554535397 +0000 UTC m=+13.849750378" watchObservedRunningTime="2026-11-01 05:02:38.554816454 +0000 UTC m=+13.850031436" Nov 01 05:02:39.170141 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.170108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:39.170334 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.170178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:39.170334 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.170184 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:39.170334 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:39.170299 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:39.170480 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:39.170420 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:39.170547 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:39.170489 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:39.331635 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.331595 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="8d7dd2a9ffd34404cb7f44c92ef7af02211558f0b7dded1183d914e2913d15cb" exitCode=0 Nov 01 05:02:39.332160 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.331690 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"8d7dd2a9ffd34404cb7f44c92ef7af02211558f0b7dded1183d914e2913d15cb"} Nov 01 05:02:39.334716 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.333705 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" event={"ID":"3ed794e8-a779-46c6-8834-78145d194d1d","Type":"ContainerStarted","Data":"66547b5b87e93e7f9d13f30e56ffc903c72b8f3062de35106d8f349e9bb0fb9c"} Nov 01 05:02:39.569270 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:39.569207 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" podStartSLOduration=2.735924497 podStartE2EDuration="13.56919059s" totalImagesPullingTime="10.833266093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:39.568745756 +0000 UTC m=+14.863960734" watchObservedRunningTime="2026-11-01 05:02:39.56919059 +0000 UTC m=+14.864405569" Nov 01 05:02:41.169863 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:41.169827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:41.169863 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:41.169864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:41.170705 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:41.169825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:41.170705 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:41.169952 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:41.170705 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:41.170049 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:41.170705 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:41.170122 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:41.340665 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:41.340623 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="20b06c0c56cff192975a8146590cde71c72c260277d05e873e61110872eb99f1" exitCode=0 Nov 01 05:02:41.340841 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:41.340686 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"20b06c0c56cff192975a8146590cde71c72c260277d05e873e61110872eb99f1"} Nov 01 05:02:42.961434 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:42.961390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:42.961910 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:42.961484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:42.961910 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:42.961546 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:42.961910 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:42.961595 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:42.961910 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:42.961621 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret podName:d20a9070-33a8-4606-b4c4-3c38e2b4abe0 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:58.961598353 +0000 UTC m=+34.256813325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret") pod "global-pull-secret-syncer-kwpzs" (UID: "d20a9070-33a8-4606-b4c4-3c38e2b4abe0") : object "kube-system"/"original-pull-secret" not registered Nov 01 05:02:42.961910 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:42.961648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:02:58.961630991 +0000 UTC m=+34.256845958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 05:02:43.164162 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:43.164113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:43.164361 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.164324 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 05:02:43.164361 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.164352 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 05:02:43.164361 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.164365 2581 projected.go:196] Error preparing data for projected volume kube-api-access-z7lm2 for pod openshift-network-diagnostics/network-check-target-5fx5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:43.164514 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.164413 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2 podName:7c97af66-cbb9-433c-9053-24eb8896f8d3 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:59.164398853 +0000 UTC m=+34.459613809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7lm2" (UniqueName: "kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2") pod "network-check-target-5fx5x" (UID: "7c97af66-cbb9-433c-9053-24eb8896f8d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 05:02:43.170316 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:43.170223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:43.170316 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:43.170222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:43.170547 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.170457 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:43.170547 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:43.170519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:43.170642 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.170618 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:43.170723 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:43.170698 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:44.352794 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.352551 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"705fa35a58ebe7b37167db937ff0a499d09617a7af1808b8e085ebdd3710bfce"} Nov 01 05:02:44.353201 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.352859 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"364875d22acabcc1cff225126d0a286cbbc30ed89f865c77c550c25fd28cc619"} Nov 01 05:02:44.353201 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.352912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"24c90e6d9155620314f1a31188648064b3225f9edeb5f95f88b5ea8567af6531"} Nov 01 05:02:44.353201 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.352956 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"e485008c84d7e59539b8719c61016d4c256d2abfb8094bce3c637dbd77254843"} Nov 01 05:02:44.353201 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.353011 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"8fbc4f4d87b6b392ec6d060c8a4296592af16f541085c85ef80eef349071a11d"} Nov 01 05:02:44.353201 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.353056 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"5995f20297a4ef0c5f962acb5d130284906d50acf725414b5cb8b7cd3c4145c8"} Nov 01 05:02:44.356025 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.355858 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="583b2bed223ca172033e536a5de013b2dae89885f01bdc69ef023a963b4b5e3e" exitCode=0 Nov 01 05:02:44.356113 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:44.355949 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"583b2bed223ca172033e536a5de013b2dae89885f01bdc69ef023a963b4b5e3e"} Nov 01 05:02:45.170526 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:45.170490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:45.170702 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:45.170634 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:45.170702 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:45.170661 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:45.170817 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:45.170756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:45.170817 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:45.170759 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:45.170906 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:45.170876 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:45.361541 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:45.361508 2581 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="9e84d15dbe05eddc0988f024c232f33e92c824a90374c9a6349abe2302359344" exitCode=0 Nov 01 05:02:45.361931 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:45.361569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerDied","Data":"9e84d15dbe05eddc0988f024c232f33e92c824a90374c9a6349abe2302359344"} Nov 01 05:02:46.368971 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:46.368939 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rfqts" event={"ID":"61cb2aa1-f501-48da-a536-27eb86daa84e","Type":"ContainerStarted","Data":"fa557750d48c3fe551932bd92519472ab2e909c47a4a624f4052243d08c90e3f"} Nov 01 05:02:46.588628 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:46.588568 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rfqts" podStartSLOduration=7.985929258 podStartE2EDuration="20.588553555s" totalImagesPullingTime="12.602624297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:46.588190759 +0000 UTC m=+21.883405738" watchObservedRunningTime="2026-11-01 05:02:46.588553555 +0000 UTC m=+21.883768541" Nov 01 05:02:47.169693 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:47.169665 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:47.169693 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:47.169681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:47.169693 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:47.169695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:47.169896 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:47.169778 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:47.169896 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:47.169844 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:47.169896 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:47.169889 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:47.374289 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:47.374215 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"f47c70e764a1fb3778ca4006270e0ea8a1f508c40082baa75aeef6d20c67a44d"} Nov 01 05:02:49.169897 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.169853 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:49.169897 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.169874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:49.170595 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:49.170025 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:49.170595 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.170070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:49.170595 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:49.170138 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:49.170595 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:49.170187 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:49.383000 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.382966 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" event={"ID":"fa48682c-8d53-4fb0-8815-618b9720f89a","Type":"ContainerStarted","Data":"181bf46e29b5c68d4f1cf33ef4df5d1c08e5a3022b876521aa5b50750244d1c1"} Nov 01 05:02:49.383235 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.383216 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:49.399509 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.399486 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:49.658989 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:49.658937 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" podStartSLOduration=7.411451308 podStartE2EDuration="23.658923128s" totalImagesPullingTime="16.24747182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:49.65841476 +0000 UTC m=+24.953629740" watchObservedRunningTime="2026-11-01 05:02:49.658923128 +0000 UTC m=+24.954138107" Nov 01 05:02:50.388925 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:50.388869 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:50.389424 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:50.389393 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:50.403869 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:50.403845 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:02:51.170797 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.170553 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:51.170936 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.170567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:51.170936 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.170898 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:51.171032 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.170610 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:51.171032 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.170994 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:51.171134 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.171077 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:51.402812 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.402780 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kwpzs"] Nov 01 05:02:51.403182 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.402889 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:51.403182 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.402999 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:51.405731 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.405703 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fx5x"] Nov 01 05:02:51.405863 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.405798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:51.405918 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.405895 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:51.406243 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.406223 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfwcx"] Nov 01 05:02:51.406345 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:51.406331 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:51.406447 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:51.406428 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:53.170469 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:53.170435 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:53.170469 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:53.170448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:53.170912 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:53.170543 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:53.170912 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:53.170595 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:53.170912 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:53.170641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:53.170912 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:53.170768 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:55.170619 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:55.170588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:55.171141 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:55.170729 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:55.171141 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:55.170774 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tfwcx" podUID="fae2a514-e8d2-4cc6-863c-16423cb450aa" Nov 01 05:02:55.171141 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:55.170840 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwpzs" podUID="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" Nov 01 05:02:55.171141 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:55.170853 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:55.171141 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:55.170959 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fx5x" podUID="7c97af66-cbb9-433c-9053-24eb8896f8d3" Nov 01 05:02:56.175203 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:56.175167 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:56.175696 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:56.175305 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:02:56.175762 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:56.175748 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:02:57.036369 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.036343 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeReady" Nov 01 05:02:57.036519 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.036471 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 05:02:57.169857 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.169826 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c7vx5"] Nov 01 05:02:57.171534 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.171515 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.174739 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.174676 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dlksj"] Nov 01 05:02:57.174884 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.174756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:57.174884 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.174787 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:57.175011 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.174990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:57.176502 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.176485 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.223537 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.223504 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 05:02:57.223699 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.223584 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 05:02:57.223699 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.223591 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 05:02:57.223699 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.223686 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7m49c\"" Nov 01 05:02:57.226537 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.226521 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k7br\"" Nov 01 05:02:57.226624 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.226584 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 05:02:57.260916 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.260889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzrr\" (UniqueName: \"kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.261016 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.260934 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.261016 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.260969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.261088 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.261063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8qv\" (UniqueName: \"kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.261164 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.261149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.261209 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.261174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.270609 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.270587 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 05:02:57.270825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.270809 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 05:02:57.270825 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.270820 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 05:02:57.271016 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.271002 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 05:02:57.271080 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.271056 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27c4k\"" Nov 01 05:02:57.271080 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.271067 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 05:02:57.271164 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.271086 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g7ktw\"" Nov 01 05:02:57.271164 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.271097 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7kc5j\"" Nov 01 05:02:57.279007 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.278989 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c7vx5"] Nov 01 05:02:57.283292 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.283269 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dlksj"] Nov 01 05:02:57.361871 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.361841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.361871 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.361870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.361891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzrr\" (UniqueName: \"kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.361912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.361932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:57.362018 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.362070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8qv\" (UniqueName: \"kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.362096 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:57.362086 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:02:57.862064447 +0000 UTC m=+33.157279405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:02:57.362416 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.362376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.362564 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.362542 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.365766 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.365747 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.422943 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.422909 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzrr\" (UniqueName: \"kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.423069 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.422975 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8qv\" (UniqueName: \"kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.482992 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.482915 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:02:57.858503 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.858472 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c7vx5"] Nov 01 05:02:57.862095 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:57.862068 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8330fe5c_2cb6_44fa_9e5a_9526e7391a26.slice/crio-cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb WatchSource:0}: Error finding container cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb: Status 404 returned error can't find the container with id cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb Nov 01 05:02:57.865784 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:57.865761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:57.865874 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:57.865861 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:02:57.865916 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:57.865908 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:02:58.865895325 +0000 UTC m=+34.161110282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:02:58.405893 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.405854 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c7vx5" event={"ID":"8330fe5c-2cb6-44fa-9e5a-9526e7391a26","Type":"ContainerStarted","Data":"cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb"} Nov 01 05:02:58.873914 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.873878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:02:58.874178 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:58.874052 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:02:58.874178 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:58.874121 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:03:00.874103795 +0000 UTC m=+36.169318776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:02:58.975095 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.975058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:58.975275 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.975156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:02:58.975275 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:58.975233 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:02:58.975364 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:02:58.975324 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:30.975305453 +0000 UTC m=+66.270520430 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:02:58.978027 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.977993 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:58.994106 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:58.994075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:02:59.176956 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.176877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:59.180411 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.180384 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:59.300546 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.300504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:02:59.369135 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.368549 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kwpzs"] Nov 01 05:02:59.409541 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.409508 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kwpzs" event={"ID":"d20a9070-33a8-4606-b4c4-3c38e2b4abe0","Type":"ContainerStarted","Data":"f0217e9d7e4d7b84d1ba55e58585e48d67db8a720738a606d2d1f6230cbe6cc9"} Nov 01 05:02:59.677031 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:02:59.676986 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fx5x"] Nov 01 05:02:59.679958 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:02:59.679914 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c97af66_cbb9_433c_9053_24eb8896f8d3.slice/crio-84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5 WatchSource:0}: Error finding container 84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5: Status 404 returned error can't find the container with id 84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5 Nov 01 05:03:00.248693 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.248655 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8"] Nov 01 05:03:00.250958 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.250936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:00.309611 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.309578 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8"] Nov 01 05:03:00.357366 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.357334 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fkkn4\"" Nov 01 05:03:00.357366 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.357363 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 05:03:00.357584 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.357393 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 05:03:00.358332 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.358313 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2r68"] Nov 01 05:03:00.360132 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.360113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.388103 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.388067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7nqs\" (UniqueName: \"kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs\") pod \"migrator-66df5d496b-8t5g8\" (UID: \"437216ed-54d5-4bbd-b1f9-6c83ae199a79\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:00.407908 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.407878 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2r68"] Nov 01 05:03:00.411921 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.411895 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvz28\"" Nov 01 05:03:00.412626 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.411941 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 05:03:00.415182 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.415144 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fx5x" event={"ID":"7c97af66-cbb9-433c-9053-24eb8896f8d3","Type":"ContainerStarted","Data":"84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5"} Nov 01 05:03:00.416095 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.416063 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 05:03:00.416211 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.416096 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 05:03:00.420232 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.420208 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 05:03:00.488630 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.488806 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.488806 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.488806 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7nqs\" (UniqueName: \"kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs\") pod \"migrator-66df5d496b-8t5g8\" (UID: \"437216ed-54d5-4bbd-b1f9-6c83ae199a79\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:00.488959 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpp87\" (UniqueName: \"kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.488959 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.488897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.552380 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.552309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7nqs\" (UniqueName: \"kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs\") pod \"migrator-66df5d496b-8t5g8\" (UID: \"437216ed-54d5-4bbd-b1f9-6c83ae199a79\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:00.563394 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.563357 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:00.589694 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.589665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.589852 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.589728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.589852 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.589749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.589852 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.589768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.589852 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.589802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpp87\" (UniqueName: \"kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.590399 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.590372 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:03:00.590541 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.590472 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.090448646 +0000 UTC m=+36.385663650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : configmap "kube-rbac-proxy" not found Nov 01 05:03:00.590541 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.590483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.590541 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.590379 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:00.590678 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.590564 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.590678 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.590590 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.090553955 +0000 UTC m=+36.385768928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:00.658880 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.658854 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpp87\" (UniqueName: \"kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:00.892376 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.892345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:00.892582 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.892515 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:03:00.892707 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:00.892583 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:03:04.892563405 +0000 UTC m=+40.187778362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:03:00.947223 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:00.947190 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8"] Nov 01 05:03:00.953086 ip-10-0-141-97 kubenswrapper[2581]: W1101 05:03:00.953056 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437216ed_54d5_4bbd_b1f9_6c83ae199a79.slice/crio-d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987 WatchSource:0}: Error finding container d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987: Status 404 returned error can't find the container with id d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987 Nov 01 05:03:01.094676 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:01.094645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:01.094676 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:01.094686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:01.094920 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:01.094806 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:03:01.094920 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:01.094864 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:02.094847525 +0000 UTC m=+37.390062495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : configmap "kube-rbac-proxy" not found Nov 01 05:03:01.095040 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:01.094951 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:01.095040 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:01.094988 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:02.094976111 +0000 UTC m=+37.390191082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:01.422112 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:01.422077 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c7vx5" event={"ID":"8330fe5c-2cb6-44fa-9e5a-9526e7391a26","Type":"ContainerStarted","Data":"e506d4d7bc8286bfb472f1cbc340956f7c46ceff56c5616e889783da353edcdc"} Nov 01 05:03:01.424198 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:01.424168 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" event={"ID":"437216ed-54d5-4bbd-b1f9-6c83ae199a79","Type":"ContainerStarted","Data":"d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987"} Nov 01 05:03:01.542017 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:01.541951 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c7vx5" podStartSLOduration=1.5455380760000001 podStartE2EDuration="4.541931428s" totalImagesPullingTime="2.996393352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:01.540894154 +0000 UTC m=+36.836109134" watchObservedRunningTime="2026-11-01 05:03:01.541931428 +0000 UTC m=+36.837146408" Nov 01 05:03:02.102430 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:02.102388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:02.102627 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:02.102445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:02.102627 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:02.102614 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:03:02.102728 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:02.102682 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:04.102662968 +0000 UTC m=+39.397877948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : configmap "kube-rbac-proxy" not found Nov 01 05:03:02.103070 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:02.103050 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:02.103127 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:02.103109 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:04.103092753 +0000 UTC m=+39.398307713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:03.430642 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:03.430602 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" event={"ID":"437216ed-54d5-4bbd-b1f9-6c83ae199a79","Type":"ContainerStarted","Data":"622b28c9309af1547b8f6e4a3503f599bd8364d1dfcb8a20babde530c869dac6"} Nov 01 05:03:03.549212 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:03.549162 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" podStartSLOduration=1.8767728940000001 podStartE2EDuration="3.549145058s" totalImagesPullingTime="1.672372164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:03.548607535 +0000 UTC m=+38.843822514" watchObservedRunningTime="2026-11-01 05:03:03.549145058 +0000 UTC m=+38.844360037" Nov 01 05:03:04.119183 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:04.119136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:04.119401 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:04.119196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:04.119401 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:04.119330 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 05:03:04.119401 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:04.119338 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:04.119553 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:04.119414 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:08.11939275 +0000 UTC m=+43.414607721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:04.119553 ip-10-0-141-97 kubenswrapper[2581]: E1101 05:03:04.119434 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:08.119424673 +0000 UTC m=+43.414639629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : configmap "kube-rbac-proxy" not found Nov 01 05:03:04.435028 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:04.434930 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fx5x" event={"ID":"7c97af66-cbb9-433c-9053-24eb8896f8d3","Type":"ContainerStarted","Data":"55efa052a5a51ab6bf2c7afbd6fc6c8e85c2360634a40ff5f24c6144fd1905a1"} Nov 01 05:03:04.555230 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:04.555187 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5fx5x" podStartSLOduration=34.766267452 podStartE2EDuration="38.555172647s" totalImagesPullingTime="3.788905195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:04.554916347 +0000 UTC m=+39.850131327" watchObservedRunningTime="2026-11-01 05:03:04.555172647 +0000 UTC m=+39.850387626" Nov 01 05:03:04.799953 ip-10-0-141-97 kubenswrapper[2581]: I1101 05:03:04.799845 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:03:04.800009 ip-10-0-141-97 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 05:03:04.812579 ip-10-0-141-97 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 05:03:04.812745 ip-10-0-141-97 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 05:03:04.812793 ip-10-0-141-97 systemd[1]: kubelet.service: Consumed 2.379s CPU time. Nov 01 05:03:04.827789 ip-10-0-141-97 systemd[1]: Starting Kubernetes Kubelet... Nov 01 05:03:04.889429 ip-10-0-141-97 kubenswrapper[4425]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:03:04.889429 ip-10-0-141-97 kubenswrapper[4425]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 05:03:04.889429 ip-10-0-141-97 kubenswrapper[4425]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:03:04.889429 ip-10-0-141-97 kubenswrapper[4425]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 05:03:04.891465 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891390 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.891465 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891465 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891469 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891473 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891475 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891478 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891480 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891483 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891486 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891488 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891491 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891493 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891496 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891498 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891500 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891503 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891505 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891508 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891510 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891513 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891515 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.891535 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891518 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891520 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891523 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891526 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891528 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891534 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891537 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891540 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891542 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891545 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891548 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891550 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891553 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891555 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891558 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891560 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891563 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891565 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891567 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891570 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.892016 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891572 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891575 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891577 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891579 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891581 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891585 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891587 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891589 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891592 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891594 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891596 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891599 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891601 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891604 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891607 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891609 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891612 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891615 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891617 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.892561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891619 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891622 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891625 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891627 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891629 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891632 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891634 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891636 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891639 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891642 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891645 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891647 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891650 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891652 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891654 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891657 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891660 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891664 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891667 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.893025 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891671 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891674 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891676 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891679 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891682 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891684 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891686 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891693 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891697 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891701 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891704 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891707 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891709 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891712 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891714 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891717 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891720 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891722 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891725 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891727 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.893504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891730 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891732 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891735 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891737 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891740 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891743 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.891746 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892480 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892487 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892491 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892494 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892499 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892501 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892505 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892507 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892510 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892513 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892515 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892518 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.893976 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892522 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892524 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892526 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892529 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892531 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892534 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892537 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892539 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892542 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892545 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892548 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892550 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892553 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892555 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892558 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892560 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892563 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892567 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892571 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.894478 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892574 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892577 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892580 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892582 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892585 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892587 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892590 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892593 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892596 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892598 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892601 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892603 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892607 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892610 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892612 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892614 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892617 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892619 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892622 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.894933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892624 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892627 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892630 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892633 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892635 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892637 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892641 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892643 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892646 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892648 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892651 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892653 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892656 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892658 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892661 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892663 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892666 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892668 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892671 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892673 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.895409 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892675 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892678 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892681 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892685 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892687 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892690 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892693 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892695 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892698 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892700 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892703 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892706 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892708 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892711 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892713 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892715 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892718 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892720 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892722 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892725 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.895886 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892728 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892732 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892736 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892739 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892743 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892747 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892750 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892753 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892755 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892757 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892761 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892763 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892766 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892768 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892771 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.892774 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893290 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893297 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893301 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893304 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.896374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893307 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893309 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893312 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893314 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893317 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893320 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893322 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893325 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893327 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893330 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893332 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893335 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893337 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893340 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893342 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893345 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893347 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893351 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893353 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893355 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.896850 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893358 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893360 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893363 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893365 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893368 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893370 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893373 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893381 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893384 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893386 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893389 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893391 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893394 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893396 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893399 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893402 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893404 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893406 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893409 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893411 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.897437 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893413 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893416 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893418 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893421 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893423 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893426 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893429 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893433 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893435 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893438 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893441 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893444 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893446 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893449 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893451 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893454 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893456 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893459 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893462 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.897918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893466 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893469 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893471 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893474 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893476 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893479 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893482 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893484 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893486 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893489 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893492 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893494 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893497 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893499 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893502 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893504 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893507 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893510 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893512 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.898476 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893515 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893519 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893522 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893526 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893529 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893532 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893534 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893537 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893540 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893542 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893545 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893547 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893565 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893570 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893573 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893576 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893579 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893582 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893585 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.899208 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893587 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893590 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893593 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893595 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.893599 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893693 4425 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893703 4425 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893712 4425 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893717 4425 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893721 4425 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893724 4425 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893728 4425 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893732 4425 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893736 4425 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893739 4425 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893742 4425 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893745 4425 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893749 4425 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893752 4425 flags.go:64] FLAG: --cgroup-root="" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893755 4425 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893758 4425 flags.go:64] FLAG: --client-ca-file="" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893760 4425 flags.go:64] FLAG: --cloud-provider="external" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893763 4425 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893767 4425 flags.go:64] FLAG: --cluster-domain="" Nov 01 05:03:04.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893770 4425 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893773 4425 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893778 4425 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893781 4425 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893785 4425 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893789 4425 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893792 4425 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893795 4425 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893798 4425 flags.go:64] FLAG: --contention-profiling="false" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893801 4425 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893803 4425 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893806 4425 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893809 4425 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893813 4425 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893816 4425 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893819 4425 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893822 4425 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893825 4425 flags.go:64] FLAG: --enable-server="true" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893828 4425 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893832 4425 flags.go:64] FLAG: --event-burst="100" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893835 4425 flags.go:64] FLAG: --event-qps="50" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893839 4425 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893842 4425 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893845 4425 flags.go:64] FLAG: --eviction-hard="" Nov 01 05:03:04.900268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893850 4425 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893853 4425 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893856 4425 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893858 4425 flags.go:64] FLAG: --eviction-soft="" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893861 4425 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893864 4425 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893867 4425 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893870 4425 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893872 4425 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893875 4425 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893880 4425 flags.go:64] FLAG: --feature-gates="" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893884 4425 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893887 4425 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893890 4425 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893894 4425 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893897 4425 flags.go:64] FLAG: --healthz-port="10248" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893900 4425 flags.go:64] FLAG: --help="false" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893903 4425 flags.go:64] FLAG: --hostname-override="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893906 4425 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893909 4425 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893911 4425 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893915 4425 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893918 4425 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 05:03:04.900850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893921 4425 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893924 4425 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893927 4425 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893929 4425 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893932 4425 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893935 4425 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893939 4425 flags.go:64] FLAG: --kube-reserved="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893942 4425 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893945 4425 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893948 4425 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893951 4425 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893954 4425 flags.go:64] FLAG: --lock-file="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893956 4425 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893959 4425 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893962 4425 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893968 4425 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893970 4425 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893973 4425 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893976 4425 flags.go:64] FLAG: --logging-format="text" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893980 4425 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893983 4425 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893986 4425 flags.go:64] FLAG: --manifest-url="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893989 4425 flags.go:64] FLAG: --manifest-url-header="" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893994 4425 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.893997 4425 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 05:03:04.901412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894001 4425 flags.go:64] FLAG: --max-pods="110" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894004 4425 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894007 4425 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894009 4425 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894013 4425 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894015 4425 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894018 4425 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894021 4425 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894027 4425 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894030 4425 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894033 4425 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894036 4425 flags.go:64] FLAG: --pod-cidr="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894039 4425 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894043 4425 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894046 4425 flags.go:64] FLAG: --pods-per-core="0" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894049 4425 flags.go:64] FLAG: --port="10250" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894052 4425 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894055 4425 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02e3120d51be079ee" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894058 4425 flags.go:64] FLAG: --qos-reserved="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894061 4425 flags.go:64] FLAG: --read-only-port="10255" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894064 4425 flags.go:64] FLAG: --register-node="true" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894066 4425 flags.go:64] FLAG: --register-with-taints="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894070 4425 flags.go:64] FLAG: --registry-burst="10" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894073 4425 flags.go:64] FLAG: --registry-qps="5" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894075 4425 flags.go:64] FLAG: --reserved-cpus="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894078 4425 flags.go:64] FLAG: --reserved-memory="" Nov 01 05:03:04.902150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894083 4425 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894085 4425 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894088 4425 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894091 4425 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894094 4425 flags.go:64] FLAG: --runonce="false" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894097 4425 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894100 4425 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894103 4425 flags.go:64] FLAG: --seccomp-default="false" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894106 4425 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894109 4425 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894112 4425 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894115 4425 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894118 4425 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894121 4425 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894123 4425 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894126 4425 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894129 4425 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894132 4425 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894135 4425 flags.go:64] FLAG: --system-cgroups="" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894139 4425 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894144 4425 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894146 4425 flags.go:64] FLAG: --tls-cert-file="" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894149 4425 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894153 4425 flags.go:64] FLAG: --tls-min-version="" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894155 4425 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 05:03:04.902774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894158 4425 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894161 4425 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894164 4425 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894167 4425 flags.go:64] FLAG: --v="2" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894171 4425 flags.go:64] FLAG: --version="false" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894175 4425 flags.go:64] FLAG: --vmodule="" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894179 4425 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894183 4425 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894381 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894386 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894389 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894392 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894394 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894397 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894400 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894403 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894406 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894408 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894411 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894413 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894416 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.903493 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894418 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894420 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894423 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894426 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894428 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894433 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894435 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894438 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894440 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894443 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894445 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894448 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894450 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894453 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894455 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894460 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894463 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894466 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894470 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.904002 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894473 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894475 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894478 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894481 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894483 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894486 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894488 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894497 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894500 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894502 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894505 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894507 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894510 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894512 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894514 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894517 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894519 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894521 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894525 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894528 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.904536 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894530 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894536 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894538 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894540 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894543 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894545 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894547 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894550 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894553 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894555 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894557 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894561 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894563 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894566 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894568 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894571 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894573 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894576 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894578 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894581 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.905114 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894589 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894592 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894595 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894597 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894599 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894602 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894605 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894607 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894610 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894612 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894615 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894617 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894620 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894622 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894624 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894626 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894629 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894631 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894634 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894637 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.905762 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894641 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894643 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894646 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894650 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894653 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894655 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894657 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894660 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894662 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894665 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894667 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894670 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894672 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894681 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.894708 4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:03:04.906660 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894781 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894786 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894789 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894792 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894794 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894797 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894799 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894802 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894804 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894807 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894809 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894812 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894814 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894817 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894820 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894822 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894825 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894827 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894830 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.907207 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894832 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894835 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894839 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894841 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894844 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894846 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894849 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894851 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894854 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894856 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894858 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894861 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894869 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894872 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894876 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894879 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894888 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894890 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894893 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894895 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.907966 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894898 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894900 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894903 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894905 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894907 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894910 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894913 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894915 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894920 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894922 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894925 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894927 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894929 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894932 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894935 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894938 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894941 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894943 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894946 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.908473 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894948 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894951 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894953 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894956 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894958 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894961 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894970 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894973 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894975 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894978 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894980 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894983 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894985 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894988 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894990 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894993 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894995 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.894997 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895001 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.908942 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895004 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895008 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895013 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895015 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895018 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895020 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895023 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895025 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895028 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895031 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895033 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895035 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895038 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895040 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895043 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895045 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895048 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895050 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895052 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.909428 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895055 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895063 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895067 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895070 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895072 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895074 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895077 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895079 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895081 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.895084 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.899711 4425 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.899728 4425 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.899748 4425 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.899753 4425 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899810 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899815 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899818 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899821 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899824 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899827 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.909875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899830 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899832 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899835 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899839 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899842 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899845 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899848 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899851 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899853 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899858 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899864 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899868 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899871 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899873 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899876 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899878 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899881 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899883 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899886 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.910370 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899889 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899892 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899894 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899897 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899900 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899903 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899905 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899908 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899911 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899913 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899916 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899918 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899920 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899923 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899926 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899928 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899931 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899933 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899936 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899938 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.910837 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899941 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899943 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899946 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899949 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899952 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899954 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899957 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899959 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899962 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899964 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899967 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899969 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899972 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899974 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899978 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899980 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899983 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899985 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899988 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.911460 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899990 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899993 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899995 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.899997 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900000 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900002 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900005 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900007 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900009 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900012 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900014 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900017 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900019 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900022 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900024 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900027 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900030 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900033 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900035 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.911958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900038 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900040 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900043 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900046 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900049 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900051 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900054 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900056 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900059 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900062 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900064 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900067 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900069 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900071 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900074 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900076 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900079 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900082 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900084 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900086 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.912417 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900089 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900091 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900093 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.900121 4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900270 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900274 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900278 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900281 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900284 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900287 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900290 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900292 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900296 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900301 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900304 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900307 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.912885 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900310 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900312 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900315 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900317 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900320 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900322 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900325 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900327 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900330 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900332 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900334 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900337 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900339 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900342 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900345 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900347 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900350 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900352 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900355 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900357 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.913303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900360 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900362 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900364 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900367 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900369 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900372 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900375 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900377 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900379 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900382 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900385 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900387 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900390 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900392 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900396 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900400 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900402 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900405 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900408 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.913782 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900410 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900413 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900415 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900417 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900420 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900422 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900425 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900428 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900430 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900432 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900435 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900437 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900440 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900442 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900445 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900447 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900450 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900452 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900454 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900457 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.914374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900460 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900462 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900465 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900467 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900470 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900472 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900475 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900477 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900479 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900482 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900484 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900487 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900489 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900492 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900494 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900496 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900499 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900502 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.915151 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900504 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900506 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900509 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900511 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900513 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900516 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900518 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900520 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900523 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900525 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900527 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900530 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900532 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900534 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900537 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900540 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.915738 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900543 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.900566 4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900598 4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900601 4425 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900604 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900607 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900611 4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900614 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900617 4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900620 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900623 4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900626 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900629 4425 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900632 4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900635 4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 05:03:04.916121 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900638 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900641 4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900643 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900646 4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900648 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900651 4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900653 4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900656 4425 feature_gate.go:386] unrecognized feature gate: Example Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900658 4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900661 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900663 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900665 4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900668 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900670 4425 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900672 4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900675 4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900677 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900681 4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900683 4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900686 4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 05:03:04.916508 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900688 4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900690 4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900693 4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900696 4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900698 4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900700 4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900703 4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900706 4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900708 4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900711 4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900713 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900715 4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900718 4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900720 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900723 4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900725 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900728 4425 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900730 4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900733 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 05:03:04.917010 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900735 4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900737 4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900740 4425 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900742 4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900744 4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900747 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900749 4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900752 4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900754 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900760 4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900763 4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900767 4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900769 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900772 4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900774 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900777 4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900779 4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900782 4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900785 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 05:03:04.917486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900787 4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900791 4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900795 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900798 4425 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900801 4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900804 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900806 4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900809 4425 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900811 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900813 4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900816 4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900818 4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900821 4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900823 4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900826 4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900828 4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900831 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900833 4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900835 4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900838 4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 05:03:04.917970 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900840 4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900843 4425 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900846 4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900850 4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900853 4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900856 4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900859 4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900861 4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900864 4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900867 4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900869 4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900871 4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900874 4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900877 4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:04.900879 4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.900938 4425 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.901905 4425 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.901964 4425 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.902154 4425 server.go:1014] "Starting client certificate rotation" Nov 01 05:03:04.918538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.902277 4425 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.902328 4425 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 04:57:25 +0000 UTC" deadline="2028-05-24 05:49:40.704622529 +0000 UTC" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.902356 4425 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13680h46m35.802268147s" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.902617 4425 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.903681 4425 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.905210 4425 log.go:25] "Validated CRI v1 runtime API" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.907147 4425 log.go:25] "Validated CRI v1 image API" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.907919 4425 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 05:03:04.919084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.909371 4425 fs.go:135] Filesystem UUIDs: map[0c22b2a2-90d5-4af3-9781-78f12c58ecb0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7e279d0-a07c-4159-8ea9-34babcdae9e9:/dev/nvme0n1p3] Nov 01 05:03:04.919320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.909393 4425 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22a01d278c2804e1c097df26d6d22c8b79afed7fe75204eede81968f10ac7768/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22a01d278c2804e1c097df26d6d22c8b79afed7fe75204eede81968f10ac7768/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0217e9d7e4d7b84d1ba55e58585e48d67db8a720738a606d2d1f6230cbe6cc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0217e9d7e4d7b84d1ba55e58585e48d67db8a720738a606d2d1f6230cbe6cc9/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4/volumes/kubernetes.io~projected/kube-api-access-c5btn:{mountpoint:/var/lib/kubelet/pods/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4/volumes/kubernetes.io~projected/kube-api-access-c5btn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/385d84d8-4a6b-4975-b02c-a80699dd5bca/volumes/kubernetes.io~projected/kube-api-access-tbkts:{mountpoint:/var/lib/kubelet/pods/385d84d8-4a6b-4975-b02c-a80699dd5bca/volumes/kubernetes.io~projected/kube-api-access-tbkts major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~projected/kube-api-access-mpwgj:{mountpoint:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~projected/kube-api-access-mpwgj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/437216ed-54d5-4bbd-b1f9-6c83ae199a79/volumes/kubernetes.io~projected/kube-api-access-g7nqs:{mountpoint:/var/lib/kubelet/pods/437216ed-54d5-4bbd-b1f9-6c83ae199a79/volumes/kubernetes.io~projected/kube-api-access-g7nqs major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61cb2aa1-f501-48da-a536-27eb86daa84e/volumes/kubernetes.io~projected/kube-api-access-mkr95:{mountpoint:/var/lib/kubelet/pods/61cb2aa1-f501-48da-a536-27eb86daa84e/volumes/kubernetes.io~projected/kube-api-access-mkr95 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7016375e-62e0-481f-a376-91f747a787bb/volumes/kubernetes.io~projected/kube-api-access-pxskc:{mountpoint:/var/lib/kubelet/pods/7016375e-62e0-481f-a376-91f747a787bb/volumes/kubernetes.io~projected/kube-api-access-pxskc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c97af66-cbb9-433c-9053-24eb8896f8d3/volumes/kubernetes.io~projected/kube-api-access-z7lm2:{mountpoint:/var/lib/kubelet/pods/7c97af66-cbb9-433c-9053-24eb8896f8d3/volumes/kubernetes.io~projected/kube-api-access-z7lm2 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~projected/kube-api-access-nvzrr:{mountpoint:/var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~projected/kube-api-access-nvzrr major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a614c70f-fd8c-4df3-9cf9-5c287da0e33f/volumes/kubernetes.io~projected/kube-api-access-fwfq9:{mountpoint:/var/lib/kubelet/pods/a614c70f-fd8c-4df3-9cf9-5c287da0e33f/volumes/kubernetes.io~projected/kube-api-access-fwfq9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c03449c8-5c4c-44ae-87b6-d784ca1dec2e/volumes/kubernetes.io~projected/kube-api-access-qx8l8:{mountpoint:/var/lib/kubelet/pods/c03449c8-5c4c-44ae-87b6-d784ca1dec2e/volumes/kubernetes.io~projected/kube-api-access-qx8l8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39784a2-25b0-499c-9199-a4e8d3d6fc1c/volumes/kubernetes.io~projected/kube-api-access-wf8qv:{mountpoint:/var/lib/kubelet/pods/c39784a2-25b0-499c-9199-a4e8d3d6fc1c/volumes/kubernetes.io~projected/kube-api-access-wf8qv major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d20a9070-33a8-4606-b4c4-3c38e2b4abe0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d20a9070-33a8-4606-b4c4-3c38e2b4abe0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d89f9f69-4ddd-4676-a02b-d836a0cf627e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d89f9f69-4ddd-4676-a02b-d836a0cf627e/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd28d5e6-5186-4372-b177-0356bdb7166b/volumes/kubernetes.io~projected/kube-api-access-wpp87:{mountpoint:/var/lib/kubelet/pods/dd28d5e6-5186-4372-b177-0356bdb7166b/volumes/kubernetes.io~projected/kube-api-access-wpp87 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~projected/kube-api-access-r9t5h:{mountpoint:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~projected/kube-api-access-r9t5h major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fae2a514-e8d2-4cc6-863c-16423cb450aa/volumes/kubernetes.io~projected/kube-api-access-255cm:{mountpoint:/var/lib/kubelet/pods/fae2a514-e8d2-4cc6-863c-16423cb450aa/volumes/kubernetes.io~projected/kube-api-access-255cm major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e8da96623bfdea9b3b40d4e06e05fac9f821d9e6cb0232fa6e8e81e5759d8695/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d2780f6b090c71cbcb9d39c6cfba603dbbfbaa1cf468006ec38b60faee2ee600/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/6c9cce959aaed17297fd001ce772c985208ba4e4fc61cdbbf6e397bcd3688266/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c3f1c4af90df93a6e105ae95e49472af9c0e1c311e4abd0e674f17c516d98325/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/003e924164d5c6a307e42ebd3b874fad0677d296f53ebd57354bf5b78a8eb274/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8b4c54003d2e3598c856d0149fc6f0d4f962da77935b545a8e15457f210fc62b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7d6b0cf226e18abd4fe5814806692eda8d95578d0563f730ecf70e499f04bd87/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0fbbd9d9da22aad259a25cd16a166fc66c8386acd7b0fe1b63eaff440d5dd3cf/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/b9592f5dffd4d4435fc447e613254eda2e6d93dcd69d77090c049137bc8727a4/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/d36aa725ca1c08cf68d47a3ef4f6e194ea92cdbb7d681098edc43724d04036ca/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/561ee4ac603ea4058793eff2a3666f751d759af5ee4f45067ec8e14316854cfd/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/508c07cf4fa9f51eab81d8a2852b2806f4130587544b9be172e13424367efc30/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0853feb73ab9a4eb20a99405be9c9377701aeb87402b18f360d6ae6dcfdabb00/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/806bcd20d403c12b243ffa94db9695ca748bbdd399ebd95509f37ecca2640119/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/adb00070f1595bf488db3b45f72bb32648e843f1648eedae654ddb7e61b8d31c/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/bb1a0199b3515a7d0baf5ef7ca003ad7635655f39bf6286372b13dea67e3b562/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b5f6945bb9b6d2399d79872352f66a1336bda2f6cd3ac27835692ee7a4d7eea9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/827a0ffa560a79b709e33ccc872df3f9ad849662fb7c7593db5d727713d2d7fa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/61621c8aabf91165d92f4beea921d381f7d0e1d19358c66e770d46a1644a0627/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e8a2b92b16628fc38ce41cdf1aeb78ff84cd46c192c7bf26434fdd3f5a80853f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ffcd5bac6f26a81f178c42426aa4ba04e2a4ebfbc8bfd4f1c1271994c68407e6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/d33409e542b0cb555412b99c5137a42b1a2c96c3ccd79e3fb36714935ee929da/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/edef362c026261cb36473aa2e58665b4ba7efbea720ae1e1c62ae8d8ed7c8189/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/d0a929a820cbc476a1e6209a0089e54c9f3997a2473b374dfe96085ad1ef2353/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/95ce079f00da1acfbf73b8dd99d3cf9d50c990b60f7c5a50848a32506dd6d224/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/b3ae2f4f7cc9e9a01d6d008b830a58e2628e60bcac22287fdd340ba69f3d47f4/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/e88a8161ce41fc397da2fdb353975a33aca7f166d814447138d9d91c4d2aab4d/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e2dcb45100379a6415852c51eeac49a54be2c008fb5bed485f45291ac2e70e2d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/de9e5ed6c9c6ae8c9150d89ab2d7797f564b605b25311301315b4ccb4cabaa65/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/33aca212d36889b4b9b87fcc0536204b9e6604d0988c4768ca45d9fa65dbf0f7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1ecd5be3bd651880eedec6a2d92b12f5e9317cd3f780d98557188c152f214251/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/de42e77a9905c0707a59ea6d6fb2cb5f65bb166356f4c30ff36fdd6f2488c601/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/218911bbbecc7ffd908398b4b71358fffe07636e1edecbb630cc58344593ac74/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/cfdd6e3e114083e19b8d3c233c2db92dd7a1ae8827c403c990862debfa44203a/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e6889df83878535879f85000ce24eb3edb52af0dbc01d3435dacf5ec31d563b4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a6aece7e40ecbe92d2db97cd990547edb345a5b6e3fd098075e1649f847ad0e7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6308d245d8f7a44fd555c49b72cdbd5133eab685a8fa246b773a7d7b63416975/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c3de43413eabebd31d984c57fdb8b846198a6a263f7a13edb4ea9eecd65aec95/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/74d3b707d1d8c61eff4950796794ead3061f2e2a6277a3b8d2e1f41b0b431105/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.916777 4425 manager.go:217] Machine: {Timestamp:2026-11-01 05:03:04.916330333 +0000 UTC m=+0.056919242 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091150 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29feef83cd25998d16a81b61e13824 SystemUUID:ec29feef-83cd-2599-8d16-a81b61e13824 BootID:768d42b3-5193-47e8-bf6e-4180eef39daf Filesystems:[{Device:/run/containers/storage/overlay-containers/b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fae2a514-e8d2-4cc6-863c-16423cb450aa/volumes/kubernetes.io~projected/kube-api-access-255cm DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d89f9f69-4ddd-4676-a02b-d836a0cf627e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4/volumes/kubernetes.io~projected/kube-api-access-c5btn DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a614c70f-fd8c-4df3-9cf9-5c287da0e33f/volumes/kubernetes.io~projected/kube-api-access-fwfq9 DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22a01d278c2804e1c097df26d6d22c8b79afed7fe75204eede81968f10ac7768/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7016375e-62e0-481f-a376-91f747a787bb/volumes/kubernetes.io~projected/kube-api-access-pxskc DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/437216ed-54d5-4bbd-b1f9-6c83ae199a79/volumes/kubernetes.io~projected/kube-api-access-g7nqs DeviceMajor:0 DeviceMinor:216 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/385d84d8-4a6b-4975-b02c-a80699dd5bca/volumes/kubernetes.io~projected/kube-api-access-tbkts DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c39784a2-25b0-499c-9199-a4e8d3d6fc1c/volumes/kubernetes.io~projected/kube-api-access-wf8qv DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d20a9070-33a8-4606-b4c4-3c38e2b4abe0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1db90ba2a5fd4a0348235b71ad5c1bc8b714da213c80b058d22fcdff9b60992/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd28d5e6-5186-4372-b177-0356bdb7166b/volumes/kubernetes.io~projected/kube-api-access-wpp87 DeviceMajor:0 DeviceMinor:219 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~projected/kube-api-access-r9t5h DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61cb2aa1-f501-48da-a536-27eb86daa84e/volumes/kubernetes.io~projected/kube-api-access-mkr95 DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d70c55e1da9dc99659bd513c8714c4b8fe1a2a7cb112257096f51bb66e623987/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c03449c8-5c4c-44ae-87b6-d784ca1dec2e/volumes/kubernetes.io~projected/kube-api-access-qx8l8 DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e33ab0420500a9fed9a8d09b5dbb6d5dfb5fc433e49bb407d3fe44cce747490/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/850cc7215e1e945fa2fddda84d3ec89e233102d9c727647ad0c302e45333a86a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fa48682c-8d53-4fb0-8815-618b9720f89a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/132b4d0602c9d5f01dfd391b3326f9be08e8e1256f14375705281668e9d96eda/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0217e9d7e4d7b84d1ba55e58585e48d67db8a720738a606d2d1f6230cbe6cc9/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3a23f985cef536bc6ffb6823d70df77cb3e4f871cd3079d1e1937db5bf75d24/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bdd6a55d766fcf1ddaf54c034dbfc2b17abfd3fc0cdeb8e931ed999a2fe171a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf4c27c8ad4b01f57c5846f6291be7f7d70dc6b170892c676d468769fa1ac6bb/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7c97af66-cbb9-433c-9053-24eb8896f8d3/volumes/kubernetes.io~projected/kube-api-access-z7lm2 DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84ef04256022e5695325ad6cba07ba1b53059bde4413ffc5d076ad6bb5425fd5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3ed794e8-a779-46c6-8834-78145d194d1d/volumes/kubernetes.io~projected/kube-api-access-mpwgj DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b44233125ae050ab4e91ebbf53e19c1dd28c0e20252c632a67f8cc3bea706fdb/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8330fe5c-2cb6-44fa-9e5a-9526e7391a26/volumes/kubernetes.io~projected/kube-api-access-nvzrr DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:84ef04256022e56 MacAddress:fa:f1:0e:80:da:66 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:11:17:1f:b0:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:2a:81:0c:d5:7e:94 Speed:0 Mtu:8901} {Name:cf4c27c8ad4b01f MacAddress:12:72:72:6b:e2:e9 Speed:10000 Mtu:8901} {Name:d70c55e1da9dc99 MacAddress:8a:45:53:03:d5:9a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:11:17:1f:b0:33 Speed:0 Mtu:9001} {Name:f0217e9d7e4d7b8 MacAddress:9a:0b:ae:2b:33:08 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:82:db:48:06:4c:32 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:12:50:7c:7a:30:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917022 4425 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917066 4425 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917359 4425 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917384 4425 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-97.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917503 4425 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917510 4425 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917520 4425 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917532 4425 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917613 4425 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917772 4425 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917876 4425 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917909 4425 kubelet.go:493] "Attempting to sync node with API server" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917922 4425 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917932 4425 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917940 4425 kubelet.go:405] "Adding apiserver pod source" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.917947 4425 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.918517 4425 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.918525 4425 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:03:04.919483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.919027 4425 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 05:03:04.920935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.920921 4425 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 05:03:04.920970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.920936 4425 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 05:03:04.921058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921048 4425 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 05:03:04.921266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921240 4425 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 05:03:04.921312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921288 4425 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 05:03:04.921384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921374 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 05:03:04.921413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921389 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 05:03:04.921413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921396 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 05:03:04.921413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921402 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 05:03:04.921413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921407 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 05:03:04.921413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921413 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921418 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921424 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921431 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921437 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921446 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921454 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921480 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 05:03:04.921548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.921485 4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 05:03:04.924766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.924755 4425 server.go:1263] "Started kubelet" Nov 01 05:03:04.925384 ip-10-0-141-97 systemd[1]: Started Kubernetes Kubelet. Nov 01 05:03:04.925560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.925480 4425 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 05:03:04.925560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.924874 4425 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 05:03:04.925666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.925581 4425 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 05:03:04.925901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.925881 4425 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 05:03:04.928287 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.927533 4425 server.go:317] "Adding debug handlers to kubelet server" Nov 01 05:03:04.929225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.928861 4425 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 05:03:04.929225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929034 4425 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 05:03:04.929225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929083 4425 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:27 +0000 UTC" deadline="2028-05-17 08:57:34.977988273 +0000 UTC" Nov 01 05:03:04.929225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929116 4425 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13515h54m30.048875797s" Nov 01 05:03:04.929225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929203 4425 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 05:03:04.929531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929239 4425 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 05:03:04.929531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.929240 4425 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 05:03:04.929531 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:04.929276 4425 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:03:04.930842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930807 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c03449c8-5c4c-44ae-87b6-d784ca1dec2e" volumeName="kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930844 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd28d5e6-5186-4372-b177-0356bdb7166b" volumeName="kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930855 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd28d5e6-5186-4372-b177-0356bdb7166b" volumeName="kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930864 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4" volumeName="kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930876 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61cb2aa1-f501-48da-a536-27eb86daa84e" volumeName="kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930889 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa48682c-8d53-4fb0-8815-618b9720f89a" volumeName="kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930903 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39784a2-25b0-499c-9199-a4e8d3d6fc1c" volumeName="kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930912 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385d84d8-4a6b-4975-b02c-a80699dd5bca" volumeName="kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930929 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61cb2aa1-f501-48da-a536-27eb86daa84e" volumeName="kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 05:03:04.930940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930941 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7016375e-62e0-481f-a376-91f747a787bb" volumeName="kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930949 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8330fe5c-2cb6-44fa-9e5a-9526e7391a26" volumeName="kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930963 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c03449c8-5c4c-44ae-87b6-d784ca1dec2e" volumeName="kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930977 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61cb2aa1-f501-48da-a536-27eb86daa84e" volumeName="kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.930986 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c97af66-cbb9-433c-9053-24eb8896f8d3" volumeName="kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931000 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a614c70f-fd8c-4df3-9cf9-5c287da0e33f" volumeName="kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931012 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c03449c8-5c4c-44ae-87b6-d784ca1dec2e" volumeName="kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931020 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d89f9f69-4ddd-4676-a02b-d836a0cf627e" volumeName="kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931028 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa48682c-8d53-4fb0-8815-618b9720f89a" volumeName="kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931043 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fae2a514-e8d2-4cc6-863c-16423cb450aa" volumeName="kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931056 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385d84d8-4a6b-4975-b02c-a80699dd5bca" volumeName="kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931066 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61cb2aa1-f501-48da-a536-27eb86daa84e" volumeName="kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931074 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20a9070-33a8-4606-b4c4-3c38e2b4abe0" volumeName="kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931082 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d89f9f69-4ddd-4676-a02b-d836a0cf627e" volumeName="kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931092 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ed794e8-a779-46c6-8834-78145d194d1d" volumeName="kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931101 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39784a2-25b0-499c-9199-a4e8d3d6fc1c" volumeName="kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931111 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ed794e8-a779-46c6-8834-78145d194d1d" volumeName="kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931126 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ed794e8-a779-46c6-8834-78145d194d1d" volumeName="kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931137 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="437216ed-54d5-4bbd-b1f9-6c83ae199a79" volumeName="kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931152 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8330fe5c-2cb6-44fa-9e5a-9526e7391a26" volumeName="kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931164 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa48682c-8d53-4fb0-8815-618b9720f89a" volumeName="kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931174 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa48682c-8d53-4fb0-8815-618b9720f89a" volumeName="kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931182 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4" volumeName="kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931194 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7016375e-62e0-481f-a376-91f747a787bb" volumeName="kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931209 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39784a2-25b0-499c-9199-a4e8d3d6fc1c" volumeName="kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931220 4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa48682c-8d53-4fb0-8815-618b9720f89a" volumeName="kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib" seLinuxMountContext="" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931227 4425 reconstruct.go:97] "Volume reconstruction finished" Nov 01 05:03:04.931410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.931233 4425 reconciler.go:29] "Reconciler: start to sync state" Nov 01 05:03:04.933215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.933193 4425 factory.go:55] Registering systemd factory Nov 01 05:03:04.933215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.933212 4425 factory.go:223] Registration of the systemd container factory successfully Nov 01 05:03:04.933957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.933847 4425 factory.go:153] Registering CRI-O factory Nov 01 05:03:04.933957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.933866 4425 factory.go:223] Registration of the crio container factory successfully Nov 01 05:03:04.933957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.933922 4425 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 05:03:04.935313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.934285 4425 factory.go:103] Registering Raw factory Nov 01 05:03:04.935313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.934382 4425 manager.go:1199] Started watching for new ooms in manager Nov 01 05:03:04.935313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.934714 4425 manager.go:319] Starting recovery of all containers Nov 01 05:03:04.962921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.962888 4425 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 05:03:04.965427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.965406 4425 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 05:03:04.965427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.965430 4425 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 05:03:04.965623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.965455 4425 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 05:03:04.965623 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:04.965495 4425 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 05:03:04.991384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991345 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="9e84d15dbe05eddc0988f024c232f33e92c824a90374c9a6349abe2302359344" exitCode=0 Nov 01 05:03:04.991384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991382 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="583b2bed223ca172033e536a5de013b2dae89885f01bdc69ef023a963b4b5e3e" exitCode=0 Nov 01 05:03:04.991598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991396 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="20b06c0c56cff192975a8146590cde71c72c260277d05e873e61110872eb99f1" exitCode=0 Nov 01 05:03:04.991598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991406 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="8d7dd2a9ffd34404cb7f44c92ef7af02211558f0b7dded1183d914e2913d15cb" exitCode=0 Nov 01 05:03:04.991598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991416 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="355342d9cbc9384a0f579b4d3adef9411a9caad0f5fb8b3d20bb4f79eef96164" exitCode=0 Nov 01 05:03:04.991598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.991426 4425 generic.go:356] "Generic (PLEG): container finished" podID="61cb2aa1-f501-48da-a536-27eb86daa84e" containerID="89ce9078ff3370d3db1623c9a162ba28190f87374b0e651b18e051d66054945f" exitCode=0 Nov 01 05:03:04.999297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:04.999277 4425 manager.go:324] Recovery completed Nov 01 05:03:05.002781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.002753 4425 generic.go:356] "Generic (PLEG): container finished" podID="73a641920c8382c6805dc090befb3983" containerID="2a06c6e65303f6377b2e3cce9e30e418b75077ac055d7fa376351adcadabb999" exitCode=0 Nov 01 05:03:05.011924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.011910 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.013003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.012986 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.013094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.013022 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.013094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.013033 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.014293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014278 4425 cpu_manager.go:226] "Starting" policy="none" Nov 01 05:03:05.014293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014292 4425 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 05:03:05.014416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014310 4425 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 05:03:05.014416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014345 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.014525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014462 4425 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 05:03:05.014525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014477 4425 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 05:03:05.014621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014529 4425 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 05:03:05.014621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014537 4425 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 05:03:05.014621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014546 4425 policy_none.go:50] "Start" Nov 01 05:03:05.014621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014559 4425 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 05:03:05.014621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014572 4425 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 05:03:05.014835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014750 4425 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 05:03:05.014835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014761 4425 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 05:03:05.014835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.014769 4425 policy_none.go:44] "Start" Nov 01 05:03:05.015152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.015131 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.015274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.015158 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.015274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.015168 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.018759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.018736 4425 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 05:03:05.021128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021117 4425 manager.go:341] "Starting Device Plugin manager" Nov 01 05:03:05.021190 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.021140 4425 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 05:03:05.021190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021157 4425 server.go:87] "Starting device plugin registration server" Nov 01 05:03:05.021498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021487 4425 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 05:03:05.021535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021503 4425 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 05:03:05.021626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021606 4425 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 05:03:05.021753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021716 4425 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 05:03:05.021832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021760 4425 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 05:03:05.021832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021767 4425 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 05:03:05.021935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.021814 4425 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:03:05.021749483Z","UUID":"c5c08c3b-dc57-41f0-8082-eeedf6457a80","Handler":null,"Name":"","Endpoint":""} Nov 01 05:03:05.023215 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.023195 4425 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 05:03:05.023316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.023221 4425 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 05:03:05.023316 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.023234 4425 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" Nov 01 05:03:05.023316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.023277 4425 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 05:03:05.066826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.066313 4425 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal"] Nov 01 05:03:05.066958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.066933 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.067875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.067861 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.067932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.067889 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.067932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.067902 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.068201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.068190 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.068352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.068337 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.068989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.068970 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.069089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069001 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.069089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069001 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.069089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069025 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.069089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069041 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.069089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069057 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.069238 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.069226 4425 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.069343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069313 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"0733015cf0f4fb0a3c7bd42111c0f986","Type":"ContainerStarted","Data":"119f6dd11b28c54caf89b3b1d03ac300437f7366a5d9aef30480ac7db418bac1"} Nov 01 05:03:05.069373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069350 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"0733015cf0f4fb0a3c7bd42111c0f986","Type":"ContainerStarted","Data":"b9d6373d65921ecdad72f6b06db7d7b4201d89f9321a917731fc0e3b0ef15c89"} Nov 01 05:03:05.069373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069368 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerStarted","Data":"2d874eae6bb71d544fa37cdae3de5511cdcd6c96c547b8187445885302d29bac"} Nov 01 05:03:05.069430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069373 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.069430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069380 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerDied","Data":"2a06c6e65303f6377b2e3cce9e30e418b75077ac055d7fa376351adcadabb999"} Nov 01 05:03:05.069430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.069394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" event={"ID":"73a641920c8382c6805dc090befb3983","Type":"ContainerStarted","Data":"08c86c06ec3f7ca83222d14d979a7ed86540ae091a27a424a90ece53bcf0f3a5"} Nov 01 05:03:05.070043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.070028 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.070093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.070058 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.070093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.070070 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.070201 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.070190 4425 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-97.us-west-1.compute.internal\" not found" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.122065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.122024 4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 05:03:05.122963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.122948 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 05:03:05.123018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.122978 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 05:03:05.123018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.122991 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 05:03:05.123018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.123016 4425 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.135285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.135242 4425 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:03:05.135371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.135321 4425 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 05:03:05.183781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.183751 4425 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:03:05.186709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.186687 4425 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:03:05.186801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.186763 4425 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 05:03:05.229902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.229869 4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.235751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.235723 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.235863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.235759 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.235863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.235784 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.240005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.239989 4425 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.240060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.240051 4425 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.240630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.240618 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeNotReady" Nov 01 05:03:05.240681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.240651 4425 setters.go:546] "Node became not ready" node="ip-10-0-141-97.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T05:03:05Z","lastTransitionTime":"2026-11-01T05:03:05Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 05:03:05.288085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.288058 4425 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-97.us-west-1.compute.internal" is owned by different node (oldNodeID="2dbc6e3b-479c-405e-8281-67eca561df32", newNodeID=""), cleaning up... Nov 01 05:03:05.292392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.292380 4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:03:05.292451 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.292418 4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.292451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.292431 4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336523 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336559 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336584 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336630 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0733015cf0f4fb0a3c7bd42111c0f986-config\") pod \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"0733015cf0f4fb0a3c7bd42111c0f986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336639 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.336775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.336630 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a641920c8382c6805dc090befb3983-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" (UID: \"73a641920c8382c6805dc090befb3983\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.351848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.351827 4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:03:05.351926 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:05.351871 4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:05.918672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:05.918646 4425 apiserver.go:52] "Watching apiserver" Nov 01 05:03:06.005998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.005963 4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:06.006566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.006544 4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:06.028445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.028417 4425 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 05:03:06.028852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.028833 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kwpzs","openshift-insights/insights-runtime-extractor-p2r68","kube-system/konnectivity-agent-mhnv9","openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8","openshift-multus/multus-additional-cni-plugins-rfqts","kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-s6ddw","openshift-dns/dns-default-dlksj","openshift-ingress-canary/ingress-canary-c7vx5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc","openshift-dns/node-resolver-b7dxf","openshift-multus/network-metrics-daemon-tfwcx","openshift-network-diagnostics/network-check-target-5fx5x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-s7lrf","openshift-image-registry/node-ca-7b7g8","openshift-multus/multus-jz9g4","openshift-network-operator/iptables-alerter-v9bmm"] Nov 01 05:03:06.032693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.032673 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.032984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.032966 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.033656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.033639 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.040264 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040229 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:03:06.040351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040274 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:03:06.040351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040298 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:03:06.040351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040316 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.040351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040337 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.040486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.040361 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.068569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.068552 4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:03:06.068643 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.068596 4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:06.069111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.069099 4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 05:03:06.069159 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.069126 4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-97.us-west-1.compute.internal" Nov 01 05:03:06.082543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.082524 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.082892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.082879 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.084493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084476 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7kc5j\"" Nov 01 05:03:06.084716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084695 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 05:03:06.084791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084703 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tw68j\"" Nov 01 05:03:06.084791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084709 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g7ktw\"" Nov 01 05:03:06.084791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084724 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gnn2d\"" Nov 01 05:03:06.084791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.084757 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 05:03:06.085116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.085100 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 05:03:06.091339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.091325 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89f9f69-4ddd-4676-a02b-d836a0cf627e-agent-certs\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:03:06.091476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.091462 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-original-pull-secret\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.117515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.117495 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lm2\" (UniqueName: \"kubernetes.io/projected/7c97af66-cbb9-433c-9053-24eb8896f8d3-kube-api-access-z7lm2\") pod \"network-check-target-5fx5x\" (UID: \"7c97af66-cbb9-433c-9053-24eb8896f8d3\") " pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:03:06.130896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.130876 4425 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 05:03:06.140970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.140937 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwgj\" (UniqueName: \"kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.141085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.140977 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141001 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141034 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141059 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141088 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141114 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141131 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd28d5e6-5186-4372-b177-0356bdb7166b-data-volume\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141139 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141165 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141199 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141235 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141273 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.141297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141294 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141327 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141351 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141379 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141402 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141451 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141484 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8l8\" (UniqueName: \"kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.141627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141516 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-kubelet-config\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.142118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.141570 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.142173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142149 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzrr\" (UniqueName: \"kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:03:06.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142179 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142205 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.142431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142229 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.142431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142294 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxskc\" (UniqueName: \"kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.142431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142315 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142331 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.142431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142421 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142438 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142473 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142502 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-etc-tuned\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142542 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5btn\" (UniqueName: \"kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142551 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ed794e8-a779-46c6-8834-78145d194d1d-tmp\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142577 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142601 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142633 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.142689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142656 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142690 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255cm\" (UniqueName: \"kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142724 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142754 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142777 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142822 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142861 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8qv\" (UniqueName: \"kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142911 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.142945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142934 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142959 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.142985 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143006 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143021 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143047 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143078 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.143143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143121 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-tmp-dir\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143160 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143195 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143218 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143234 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143270 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpp87\" (UniqueName: \"kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143335 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385d84d8-4a6b-4975-b02c-a80699dd5bca-tmp-dir\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.143364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143336 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7016375e-62e0-481f-a376-91f747a787bb-iptables-alerter-script\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143372 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143391 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143484 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143502 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143553 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143578 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143597 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143628 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143652 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.143669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143667 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.144000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143743 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwfq9\" (UniqueName: \"kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.144000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143791 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.144000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.143812 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.144219 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144190 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9t5h\" (UniqueName: \"kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.144356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144330 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.144411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144369 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.144411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144403 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.144508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144436 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.144508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144469 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.144602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144500 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.144602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144551 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkr95\" (UniqueName: \"kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.144686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144603 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.144686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144678 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.144810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144794 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.144877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144830 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkts\" (UniqueName: \"kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.144877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.144864 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7nqs\" (UniqueName: \"kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs\") pod \"migrator-66df5d496b-8t5g8\" (UID: \"437216ed-54d5-4bbd-b1f9-6c83ae199a79\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:06.145088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145071 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.145149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145130 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.145465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145448 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.145529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145504 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.145574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145541 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.145616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145598 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.145660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145629 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.145710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145671 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.145847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.145831 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d20a9070-33a8-4606-b4c4-3c38e2b4abe0-dbus\") pod \"global-pull-secret-syncer-kwpzs\" (UID: \"d20a9070-33a8-4606-b4c4-3c38e2b4abe0\") " pod="kube-system/global-pull-secret-syncer-kwpzs" Nov 01 05:03:06.184904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.184840 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 05:03:06.185159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185143 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 05:03:06.185475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185397 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.185546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185397 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 05:03:06.185659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185589 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 05:03:06.185718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185665 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.185718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185667 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27c4k\"" Nov 01 05:03:06.185718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185690 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185703 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 05:03:06.185718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185707 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185721 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185739 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185740 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9csn\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185753 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185767 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185710 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185783 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185791 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185798 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185844 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185855 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185865 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185882 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185895 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7m49c\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185918 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nkwq4\"" Nov 01 05:03:06.185921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185929 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185946 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185951 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185951 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185967 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185960 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185987 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185991 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186001 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k7br\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.185998 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbhl7\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186054 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186111 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186111 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fhbr8\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186123 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6ndpw\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186131 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fkkn4\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186139 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186120 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvz28\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186164 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186216 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186294 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q6tm4\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186326 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186388 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 05:03:06.186952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.186484 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 05:03:06.187630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.187619 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vjvv\"" Nov 01 05:03:06.190865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.190850 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89f9f69-4ddd-4676-a02b-d836a0cf627e-konnectivity-ca\") pod \"konnectivity-agent-mhnv9\" (UID: \"d89f9f69-4ddd-4676-a02b-d836a0cf627e\") " pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:03:06.191985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.191972 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.192043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.192027 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-config-volume\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.192133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.192097 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.192205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.192191 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-daemon-config\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.193305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.193287 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-cert\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:03:06.193349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.193329 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-config\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.193378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.193348 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-serviceca\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.194348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.194330 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa48682c-8d53-4fb0-8815-618b9720f89a-ovn-node-metrics-cert\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.194397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.194342 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-env-overrides\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.194429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.194398 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa48682c-8d53-4fb0-8815-618b9720f89a-ovnkube-script-lib\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.194429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.194403 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61cb2aa1-f501-48da-a536-27eb86daa84e-cni-binary-copy\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.196636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.196612 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cni-binary-copy\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.200896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.200873 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwgj\" (UniqueName: \"kubernetes.io/projected/3ed794e8-a779-46c6-8834-78145d194d1d-kube-api-access-mpwgj\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.205676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.205644 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzrr\" (UniqueName: \"kubernetes.io/projected/8330fe5c-2cb6-44fa-9e5a-9526e7391a26-kube-api-access-nvzrr\") pod \"ingress-canary-c7vx5\" (UID: \"8330fe5c-2cb6-44fa-9e5a-9526e7391a26\") " pod="openshift-ingress-canary/ingress-canary-c7vx5" Nov 01 05:03:06.205794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.205772 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8l8\" (UniqueName: \"kubernetes.io/projected/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-kube-api-access-qx8l8\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.205876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.205847 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwfq9\" (UniqueName: \"kubernetes.io/projected/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kube-api-access-fwfq9\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.205937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.205919 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkts\" (UniqueName: \"kubernetes.io/projected/385d84d8-4a6b-4975-b02c-a80699dd5bca-kube-api-access-tbkts\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.205991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.205930 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxskc\" (UniqueName: \"kubernetes.io/projected/7016375e-62e0-481f-a376-91f747a787bb-kube-api-access-pxskc\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.206089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.206070 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8qv\" (UniqueName: \"kubernetes.io/projected/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-kube-api-access-wf8qv\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.206148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.206099 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255cm\" (UniqueName: \"kubernetes.io/projected/fae2a514-e8d2-4cc6-863c-16423cb450aa-kube-api-access-255cm\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.206354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.206338 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9t5h\" (UniqueName: \"kubernetes.io/projected/fa48682c-8d53-4fb0-8815-618b9720f89a-kube-api-access-r9t5h\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.206413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.206347 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7nqs\" (UniqueName: \"kubernetes.io/projected/437216ed-54d5-4bbd-b1f9-6c83ae199a79-kube-api-access-g7nqs\") pod \"migrator-66df5d496b-8t5g8\" (UID: \"437216ed-54d5-4bbd-b1f9-6c83ae199a79\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8t5g8" Nov 01 05:03:06.208068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.208051 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkr95\" (UniqueName: \"kubernetes.io/projected/61cb2aa1-f501-48da-a536-27eb86daa84e-kube-api-access-mkr95\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.208122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.208071 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpp87\" (UniqueName: \"kubernetes.io/projected/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-api-access-wpp87\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.235043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.235015 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5btn\" (UniqueName: \"kubernetes.io/projected/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-kube-api-access-c5btn\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.245912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245891 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.245958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245916 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.245958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245933 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.245958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245947 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245971 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245986 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7016375e-62e0-481f-a376-91f747a787bb-host-slash\") pod \"iptables-alerter-v9bmm\" (UID: \"7016375e-62e0-481f-a376-91f747a787bb\") " pod="openshift-network-operator/iptables-alerter-v9bmm" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.245993 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246026 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-etc-kubernetes\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246026 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246045 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-sys-fs\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246058 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246052 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-cnibin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246065 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246069 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-netns\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.246092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246089 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246117 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-conf-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246142 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246166 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-var-lib-kubelet\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246173 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246194 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-bin\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246201 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246203 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246244 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246290 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246294 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-lib-modules\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246300 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-netns\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246313 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246340 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246362 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.246382 4425 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246399 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246420 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-kubelet\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246426 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246439 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-hostroot\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246450 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246423 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd28d5e6-5186-4372-b177-0356bdb7166b-crio-socket\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.246486 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:03:06.746459291 +0000 UTC m=+1.887048207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246457 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246504 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246526 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246529 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-modprobe-d\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246552 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-sys\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246568 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246579 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysctl-conf\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246574 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246596 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-k8s-cni-cncf-io\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.246601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246616 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246639 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246657 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-var-lib-cni-multus\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246666 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246683 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-slash\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246691 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246705 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-etc-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246715 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246734 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-multus-socket-dir-parent\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246739 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246763 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-registration-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246764 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246787 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-kubelet\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246794 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246846 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246854 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-ovn\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246859 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-kubernetes\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246872 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246885 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-socket-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246897 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-systemd\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246896 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246920 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246918 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-system-cni-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246935 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-var-lib-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246943 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246965 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.246987 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247015 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/385d84d8-4a6b-4975-b02c-a80699dd5bca-hosts-file\") pod \"node-resolver-b7dxf\" (UID: \"385d84d8-4a6b-4975-b02c-a80699dd5bca\") " pod="openshift-dns/node-resolver-b7dxf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247019 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247040 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4-host\") pod \"node-ca-7b7g8\" (UID: \"36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4\") " pod="openshift-image-registry/node-ca-7b7g8" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247042 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247061 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247077 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247094 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-cnibin\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247020 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-etc-sysconfig\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247102 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247123 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-bin\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247131 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247158 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.247184 4425 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247132 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61cb2aa1-f501-48da-a536-27eb86daa84e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rfqts\" (UID: \"61cb2aa1-f501-48da-a536-27eb86daa84e\") " pod="openshift-multus/multus-additional-cni-plugins-rfqts" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247194 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247220 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.247233 4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.247240 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:06.747224098 +0000 UTC m=+1.887813013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247239 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-systemd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247271 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-node-log\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247286 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.247308 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:06.747293526 +0000 UTC m=+1.887882441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247312 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-host-cni-netd\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247338 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247354 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247377 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247393 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-log-socket\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247400 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247423 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247440 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247450 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-host\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247455 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a614c70f-fd8c-4df3-9cf9-5c287da0e33f-device-dir\") pod \"aws-ebs-csi-driver-node-qg6cc\" (UID: \"a614c70f-fd8c-4df3-9cf9-5c287da0e33f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg6cc" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247451 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ed794e8-a779-46c6-8834-78145d194d1d-run\") pod \"tuned-s6ddw\" (UID: \"3ed794e8-a779-46c6-8834-78145d194d1d\") " pod="openshift-cluster-node-tuning-operator/tuned-s6ddw" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247458 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247489 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-systemd-units\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247494 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247503 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-host-run-multus-certs\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247514 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa48682c-8d53-4fb0-8815-618b9720f89a-run-openvswitch\") pod \"ovnkube-node-s7lrf\" (UID: \"fa48682c-8d53-4fb0-8815-618b9720f89a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247549 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03449c8-5c4c-44ae-87b6-d784ca1dec2e-system-cni-dir\") pod \"multus-jz9g4\" (UID: \"c03449c8-5c4c-44ae-87b6-d784ca1dec2e\") " pod="openshift-multus/multus-jz9g4" Nov 01 05:03:06.247743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.247605 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd28d5e6-5186-4372-b177-0356bdb7166b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.750886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.750847 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.750912 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:06.750939 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751011 4425 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751033 4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751011 4425 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:03:06.751093 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751084 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:07.751068116 +0000 UTC m=+2.891657027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:06.751340 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751100 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:07.751091956 +0000 UTC m=+2.891680853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:03:06.751340 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:06.751123 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:03:07.751116453 +0000 UTC m=+2.891705350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:03:07.756617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:07.756576 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:07.756617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:07.756618 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:07.756649 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756724 4425 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756729 4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756798 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:09.756784372 +0000 UTC m=+4.897373269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756728 4425 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756837 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:09.75682342 +0000 UTC m=+4.897412520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:07.757065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:07.756881 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls podName:c39784a2-25b0-499c-9199-a4e8d3d6fc1c nodeName:}" failed. No retries permitted until 2026-11-01 05:03:09.756871581 +0000 UTC m=+4.897460491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls") pod "dns-default-dlksj" (UID: "c39784a2-25b0-499c-9199-a4e8d3d6fc1c") : secret "dns-default-metrics-tls" not found Nov 01 05:03:08.899535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:08.899467 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b7dxf_385d84d8-4a6b-4975-b02c-a80699dd5bca/dns-node-resolver/0.log" Nov 01 05:03:09.024901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.024874 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:03:09.407203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.407176 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:09.699277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.699192 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7b7g8_36f0ce12-35eb-4c8b-9cfc-aaba8f41ded4/node-ca/0.log" Nov 01 05:03:09.770044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.770011 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:09.770217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.770071 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:09.770217 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:09.770166 4425 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:03:09.770217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.770186 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:09.770217 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:09.770213 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:13.770199521 +0000 UTC m=+8.910788418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:03:09.770457 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:09.770275 4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:09.770457 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:09.770368 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:13.770356678 +0000 UTC m=+8.910945576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:09.770559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.770491 4425 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 05:03:09.772727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.772708 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c39784a2-25b0-499c-9199-a4e8d3d6fc1c-metrics-tls\") pod \"dns-default-dlksj\" (UID: \"c39784a2-25b0-499c-9199-a4e8d3d6fc1c\") " pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:09.823060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.823031 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:09.933895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:09.933862 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:10.015060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:10.015034 4425 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 05:03:10.308126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:10.308054 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dlksj"] Nov 01 05:03:10.311411 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:10.311379 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39784a2_25b0_499c_9199_a4e8d3d6fc1c.slice/crio-23932daf4ae664d6e0a4e240da76199634a108ef0a50bf9373d9fc1100b92db9 WatchSource:0}: Error finding container 23932daf4ae664d6e0a4e240da76199634a108ef0a50bf9373d9fc1100b92db9: Status 404 returned error can't find the container with id 23932daf4ae664d6e0a4e240da76199634a108ef0a50bf9373d9fc1100b92db9 Nov 01 05:03:10.313552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:10.313532 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:03:10.499673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:10.499645 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c7vx5_8330fe5c-2cb6-44fa-9e5a-9526e7391a26/serve-healthcheck-canary/0.log" Nov 01 05:03:11.018731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:11.018696 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlksj" event={"ID":"c39784a2-25b0-499c-9199-a4e8d3d6fc1c","Type":"ContainerStarted","Data":"23932daf4ae664d6e0a4e240da76199634a108ef0a50bf9373d9fc1100b92db9"} Nov 01 05:03:11.099637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:11.099586 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8t5g8_437216ed-54d5-4bbd-b1f9-6c83ae199a79/migrator/0.log" Nov 01 05:03:13.026089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.026057 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlksj" event={"ID":"c39784a2-25b0-499c-9199-a4e8d3d6fc1c","Type":"ContainerStarted","Data":"a897f8ff6a608be70b4a6f85a18cd140930f3d780192104c71b1d1d16c5a62d7"} Nov 01 05:03:13.026464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.026113 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dlksj" event={"ID":"c39784a2-25b0-499c-9199-a4e8d3d6fc1c","Type":"ContainerStarted","Data":"4c07765616c10cf363dfbc30093fee73dd8642a305b261a161cb2d327a2e357c"} Nov 01 05:03:13.026464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.026155 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:13.236061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.236035 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7lrf" Nov 01 05:03:13.794779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.794728 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:13.794779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:13.794778 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:13.794984 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:13.794900 4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 05:03:13.794984 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:13.794899 4425 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 05:03:13.794984 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:13.794972 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs podName:fae2a514-e8d2-4cc6-863c-16423cb450aa nodeName:}" failed. No retries permitted until 2026-11-01 05:03:21.79495286 +0000 UTC m=+16.935541763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs") pod "network-metrics-daemon-tfwcx" (UID: "fae2a514-e8d2-4cc6-863c-16423cb450aa") : secret "metrics-daemon-secret" not found Nov 01 05:03:13.795080 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:13.794986 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls podName:dd28d5e6-5186-4372-b177-0356bdb7166b nodeName:}" failed. No retries permitted until 2026-11-01 05:03:21.794980366 +0000 UTC m=+16.935569262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2r68" (UID: "dd28d5e6-5186-4372-b177-0356bdb7166b") : secret "insights-runtime-extractor-tls" not found Nov 01 05:03:15.590450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:15.590420 4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-97.us-west-1.compute.internal" event="NodeReady" Nov 01 05:03:21.850205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.850157 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:21.850205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.850223 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:21.852621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.852596 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd28d5e6-5186-4372-b177-0356bdb7166b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2r68\" (UID: \"dd28d5e6-5186-4372-b177-0356bdb7166b\") " pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:21.852741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.852657 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae2a514-e8d2-4cc6-863c-16423cb450aa-metrics-certs\") pod \"network-metrics-daemon-tfwcx\" (UID: \"fae2a514-e8d2-4cc6-863c-16423cb450aa\") " pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:21.933057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.933031 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tfwcx" Nov 01 05:03:21.934116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.934099 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2r68" Nov 01 05:03:21.978698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:21.978672 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9b96bb796-dq65h"] Nov 01 05:03:22.003310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.002420 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.043549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.043470 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b96bb796-dq65h"] Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051805 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-tls\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051857 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-trusted-ca\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051877 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-installation-pull-secrets\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051896 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-image-registry-private-configuration\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051924 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8x92\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-kube-api-access-s8x92\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.051942 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-bound-sa-token\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.052067 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-certificates\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.052586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.052096 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-ca-trust-extracted\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.152785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152696 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-certificates\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.152785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152732 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-ca-trust-extracted\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.152785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152754 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-tls\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.152785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152775 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-trusted-ca\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152799 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-installation-pull-secrets\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152908 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-image-registry-private-configuration\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.152974 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8x92\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-kube-api-access-s8x92\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.153008 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-bound-sa-token\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.153207 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-ca-trust-extracted\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.153691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.153669 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-certificates\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.181692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.181662 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 05:03:22.184617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.184598 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-trusted-ca\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.195379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.195355 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 05:03:22.195458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.195378 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 05:03:22.195458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.195427 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 05:03:22.195541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.195476 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n5nx4\"" Nov 01 05:03:22.206111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.206087 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-registry-tls\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.206325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.206305 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-installation-pull-secrets\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.206389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.206326 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-image-registry-private-configuration\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.220457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.220425 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-bound-sa-token\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.220568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.220481 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8x92\" (UniqueName: \"kubernetes.io/projected/6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf-kube-api-access-s8x92\") pod \"image-registry-9b96bb796-dq65h\" (UID: \"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf\") " pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.322536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.322506 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tfwcx"] Nov 01 05:03:22.325589 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:22.325559 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae2a514_e8d2_4cc6_863c_16423cb450aa.slice/crio-4132d66876d9612401ef0751cba615a757b5be917897ac2530ad247d2b96fd63 WatchSource:0}: Error finding container 4132d66876d9612401ef0751cba615a757b5be917897ac2530ad247d2b96fd63: Status 404 returned error can't find the container with id 4132d66876d9612401ef0751cba615a757b5be917897ac2530ad247d2b96fd63 Nov 01 05:03:22.329123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.329105 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:22.377273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.373758 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2r68"] Nov 01 05:03:22.738902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:22.738819 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b96bb796-dq65h"] Nov 01 05:03:22.741812 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:22.741786 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd44198_af2c_4ca0_8cc6_7b8ff7514fdf.slice/crio-e87399c9d4ebbd6422155e2df47bbc75754beca4939908576a2cbfea12f5a51f WatchSource:0}: Error finding container e87399c9d4ebbd6422155e2df47bbc75754beca4939908576a2cbfea12f5a51f: Status 404 returned error can't find the container with id e87399c9d4ebbd6422155e2df47bbc75754beca4939908576a2cbfea12f5a51f Nov 01 05:03:23.031033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.030966 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dlksj" Nov 01 05:03:23.056115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.056079 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" event={"ID":"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf","Type":"ContainerStarted","Data":"68214a3e87d3fcffc69873a8f5426c5d281a06a0eb461905c822bbc7114ce0be"} Nov 01 05:03:23.056320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.056146 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" event={"ID":"6dd44198-af2c-4ca0-8cc6-7b8ff7514fdf","Type":"ContainerStarted","Data":"e87399c9d4ebbd6422155e2df47bbc75754beca4939908576a2cbfea12f5a51f"} Nov 01 05:03:23.056320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.056180 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:23.058076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.057798 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2r68" event={"ID":"dd28d5e6-5186-4372-b177-0356bdb7166b","Type":"ContainerStarted","Data":"c5396f390bbc22b24b66fa7f284b2c91881c48c46bda49ee586a3e44354c7d0a"} Nov 01 05:03:23.058076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.057855 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2r68" event={"ID":"dd28d5e6-5186-4372-b177-0356bdb7166b","Type":"ContainerStarted","Data":"bbcfa47200c8d4b96d693f8b20fca19aa74a060036a04cd9aa511275f80d6a8b"} Nov 01 05:03:23.059241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.059220 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfwcx" event={"ID":"fae2a514-e8d2-4cc6-863c-16423cb450aa","Type":"ContainerStarted","Data":"4132d66876d9612401ef0751cba615a757b5be917897ac2530ad247d2b96fd63"} Nov 01 05:03:23.272047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:23.271979 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" podStartSLOduration=2.271959478 podStartE2EDuration="2.271959478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:23.271867455 +0000 UTC m=+18.412456375" watchObservedRunningTime="2026-11-01 05:03:23.271959478 +0000 UTC m=+18.412548379" Nov 01 05:03:25.067362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:25.067317 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfwcx" event={"ID":"fae2a514-e8d2-4cc6-863c-16423cb450aa","Type":"ContainerStarted","Data":"5f2402552f00486dba82f80b70aaf3dd82b29b5b0928fe9af53c37377bcae8b4"} Nov 01 05:03:25.067831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:25.067386 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tfwcx" event={"ID":"fae2a514-e8d2-4cc6-863c-16423cb450aa","Type":"ContainerStarted","Data":"074832ffad9c8e817e05099a3d4671ff10bf7e84b94be471c4e2234b0cb55790"} Nov 01 05:03:25.069358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:25.069333 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2r68" event={"ID":"dd28d5e6-5186-4372-b177-0356bdb7166b","Type":"ContainerStarted","Data":"3be41d003a6c7eaa51d022a86453adbf9aa0dfa3c3aa98036cdaf939fff1c95c"} Nov 01 05:03:26.074998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.074956 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2r68" event={"ID":"dd28d5e6-5186-4372-b177-0356bdb7166b","Type":"ContainerStarted","Data":"da8a47c42b268c78d2ba334b73d9aee4160bae49728b940e815334bf9a0613f2"} Nov 01 05:03:26.212290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.212196 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-trvcp"] Nov 01 05:03:26.213989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.213974 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.267591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267568 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 05:03:26.267789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267627 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 05:03:26.267789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267632 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 05:03:26.267878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267821 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q9rzx\"" Nov 01 05:03:26.267878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267827 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 05:03:26.267941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.267899 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 05:03:26.273996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.273976 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-trvcp"] Nov 01 05:03:26.283302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.283277 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.283426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.283303 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.283426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.283347 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a68393e-64c6-46d6-9cb5-03a2d1324c60-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.283533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.283464 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mt42\" (UniqueName: \"kubernetes.io/projected/9a68393e-64c6-46d6-9cb5-03a2d1324c60-kube-api-access-9mt42\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.383848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.383819 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a68393e-64c6-46d6-9cb5-03a2d1324c60-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.384006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.383857 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mt42\" (UniqueName: \"kubernetes.io/projected/9a68393e-64c6-46d6-9cb5-03a2d1324c60-kube-api-access-9mt42\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.384006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.383886 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.384006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.383925 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.385422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.385394 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a68393e-64c6-46d6-9cb5-03a2d1324c60-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.388784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.387402 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.388784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.387607 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a68393e-64c6-46d6-9cb5-03a2d1324c60-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.446165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.446133 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mt42\" (UniqueName: \"kubernetes.io/projected/9a68393e-64c6-46d6-9cb5-03a2d1324c60-kube-api-access-9mt42\") pod \"prometheus-operator-9cd89b74c-trvcp\" (UID: \"9a68393e-64c6-46d6-9cb5-03a2d1324c60\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.524198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.524118 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" Nov 01 05:03:26.893042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:26.893012 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-trvcp"] Nov 01 05:03:26.896081 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:26.896057 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a68393e_64c6_46d6_9cb5_03a2d1324c60.slice/crio-b8dc1d7fe99f6786fc53b99f6bd14e1daaf99f55a3a4c23870e346e028e345a8 WatchSource:0}: Error finding container b8dc1d7fe99f6786fc53b99f6bd14e1daaf99f55a3a4c23870e346e028e345a8: Status 404 returned error can't find the container with id b8dc1d7fe99f6786fc53b99f6bd14e1daaf99f55a3a4c23870e346e028e345a8 Nov 01 05:03:26.945992 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:26.945961 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:03:27.079543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:27.079505 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" event={"ID":"9a68393e-64c6-46d6-9cb5-03a2d1324c60","Type":"ContainerStarted","Data":"b8dc1d7fe99f6786fc53b99f6bd14e1daaf99f55a3a4c23870e346e028e345a8"} Nov 01 05:03:28.324506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:28.324477 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mhnv9" Nov 01 05:03:29.088948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:29.088916 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" event={"ID":"9a68393e-64c6-46d6-9cb5-03a2d1324c60","Type":"ContainerStarted","Data":"c3836a5ccfeef8039c08622a1ef191509a0f42d40041f8abb32fd32ba0c9274a"} Nov 01 05:03:29.089089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:29.088968 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" event={"ID":"9a68393e-64c6-46d6-9cb5-03a2d1324c60","Type":"ContainerStarted","Data":"78dbc7fc01465ba0686f72c711dac480dc0491aebc32ccb79461ee90046c6393"} Nov 01 05:03:29.208080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:29.208014 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-trvcp" podStartSLOduration=1.427480976 podStartE2EDuration="3.207997909s" totalImagesPullingTime="1.780516933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:29.207476837 +0000 UTC m=+24.348065756" watchObservedRunningTime="2026-11-01 05:03:29.207997909 +0000 UTC m=+24.348586825" Nov 01 05:03:31.777120 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.777081 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2"] Nov 01 05:03:31.779068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.779054 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.822219 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.822190 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.822382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.822275 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.822382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.822320 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.822450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.822397 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wj28\" (UniqueName: \"kubernetes.io/projected/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-kube-api-access-5wj28\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.883191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.883157 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 05:03:31.883191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.883157 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4t8cz\"" Nov 01 05:03:31.883417 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.883246 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 05:03:31.884119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.884101 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2"] Nov 01 05:03:31.923129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.923099 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.923129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.923129 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.923375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.923154 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wj28\" (UniqueName: \"kubernetes.io/projected/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-kube-api-access-5wj28\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.924829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.923717 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.924829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.924421 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.928768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.928737 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.929615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.929585 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:31.935388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.935367 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zsvch"] Nov 01 05:03:31.937429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.937412 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:31.985959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:31.985927 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wj28\" (UniqueName: \"kubernetes.io/projected/d5ddfea2-3759-4ad2-9b50-48acaf309f1f-kube-api-access-5wj28\") pod \"openshift-state-metrics-7bd7489957-rf9s2\" (UID: \"d5ddfea2-3759-4ad2-9b50-48acaf309f1f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:32.008614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.008587 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 05:03:32.024893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.024861 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxqh\" (UniqueName: \"kubernetes.io/projected/048500be-6716-42e5-93e3-e3b7e3507882-kube-api-access-9xxqh\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.024893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.024896 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-sys\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.024913 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-root\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.024943 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.025087 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-textfile\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.025127 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-metrics-client-ca\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.025186 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-wtmp\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.025274 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.025362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.025316 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-tls\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.052492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.052419 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 05:03:32.052660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.052537 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 05:03:32.052660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.052595 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t567f\"" Nov 01 05:03:32.090300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.090244 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" Nov 01 05:03:32.127350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127313 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-textfile\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127352 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-metrics-client-ca\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127507 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-wtmp\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127566 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127607 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-tls\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127642 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxqh\" (UniqueName: \"kubernetes.io/projected/048500be-6716-42e5-93e3-e3b7e3507882-kube-api-access-9xxqh\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127673 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-sys\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127700 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-textfile\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127700 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-root\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127762 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.127897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127779 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-root\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.128004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127919 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-metrics-client-ca\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.128004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127987 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-sys\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.128099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.127920 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-wtmp\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.128176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.128153 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.130216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.130193 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.130376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.130359 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/048500be-6716-42e5-93e3-e3b7e3507882-node-exporter-tls\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.189967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.189930 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxqh\" (UniqueName: \"kubernetes.io/projected/048500be-6716-42e5-93e3-e3b7e3507882-kube-api-access-9xxqh\") pod \"node-exporter-zsvch\" (UID: \"048500be-6716-42e5-93e3-e3b7e3507882\") " pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.249025 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.248994 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsvch" Nov 01 05:03:32.258591 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:32.258568 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048500be_6716_42e5_93e3_e3b7e3507882.slice/crio-588f8647f08c87302437f5a1dd73b53166b26ae525a61b11d44eedb65126cc83 WatchSource:0}: Error finding container 588f8647f08c87302437f5a1dd73b53166b26ae525a61b11d44eedb65126cc83: Status 404 returned error can't find the container with id 588f8647f08c87302437f5a1dd73b53166b26ae525a61b11d44eedb65126cc83 Nov 01 05:03:32.506934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:32.506900 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2"] Nov 01 05:03:32.509604 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:32.509582 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ddfea2_3759_4ad2_9b50_48acaf309f1f.slice/crio-a6cb1b5fd92b2bfe2f9bc4998a0b980d162092cdd0ee36541aabdd8095f05dee WatchSource:0}: Error finding container a6cb1b5fd92b2bfe2f9bc4998a0b980d162092cdd0ee36541aabdd8095f05dee: Status 404 returned error can't find the container with id a6cb1b5fd92b2bfe2f9bc4998a0b980d162092cdd0ee36541aabdd8095f05dee Nov 01 05:03:33.102273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:33.102184 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsvch" event={"ID":"048500be-6716-42e5-93e3-e3b7e3507882","Type":"ContainerStarted","Data":"588f8647f08c87302437f5a1dd73b53166b26ae525a61b11d44eedb65126cc83"} Nov 01 05:03:33.104043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:33.104021 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" event={"ID":"d5ddfea2-3759-4ad2-9b50-48acaf309f1f","Type":"ContainerStarted","Data":"f9d5ff8904d7af3f6f0211995aaf347e5cf936142e05e6ab6676148c0edb1e2d"} Nov 01 05:03:33.104166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:33.104064 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" event={"ID":"d5ddfea2-3759-4ad2-9b50-48acaf309f1f","Type":"ContainerStarted","Data":"9f1dea8b458f2d151a6b79983ebeeafee63e53cc5d698108f324e96b8f54265c"} Nov 01 05:03:33.104166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:33.104083 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" event={"ID":"d5ddfea2-3759-4ad2-9b50-48acaf309f1f","Type":"ContainerStarted","Data":"a6cb1b5fd92b2bfe2f9bc4998a0b980d162092cdd0ee36541aabdd8095f05dee"} Nov 01 05:03:34.108450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.108423 4425 generic.go:356] "Generic (PLEG): container finished" podID="048500be-6716-42e5-93e3-e3b7e3507882" containerID="22a04ed52f4142d17e6adae059ad1411a7bf2cfcc70cac084bef9df3100effb2" exitCode=0 Nov 01 05:03:34.108813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.108507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsvch" event={"ID":"048500be-6716-42e5-93e3-e3b7e3507882","Type":"ContainerDied","Data":"22a04ed52f4142d17e6adae059ad1411a7bf2cfcc70cac084bef9df3100effb2"} Nov 01 05:03:34.176054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.176026 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:03:34.178553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.178533 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.227485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.227461 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:03:34.246247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246178 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246210 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246296 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246340 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246389 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7prx\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246477 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246633 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.246998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.246973 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.247297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.247225 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.247382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.247333 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.247445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.247427 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.247794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.247767 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.252612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.252587 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 05:03:34.252729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.252630 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 05:03:34.331948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.331860 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 05:03:34.331948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.331860 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 05:03:34.343988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.343958 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 05:03:34.344163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.343995 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 05:03:34.344163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.344074 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 05:03:34.344163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.344132 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 05:03:34.344163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.344162 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 05:03:34.344361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.344340 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8mzd7\"" Nov 01 05:03:34.349658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349638 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349683 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349721 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349750 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349775 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349799 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.349868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349826 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349883 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349928 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349954 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.349981 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.350006 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.350055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.350037 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7prx\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.351353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.351277 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.351353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.351342 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.351353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.351389 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.353384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.353320 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.353579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.353549 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.353801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.353775 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.353916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.353901 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.353982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.353961 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.354096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.354078 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.354161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.354112 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.354989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.354971 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.355175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.355158 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.411828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.411796 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7prx\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\") pod \"alertmanager-main-0\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.512575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.512546 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:03:34.939904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:34.939697 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:03:34.942191 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:34.942161 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b649fc4_8d29_4ebd_b7d8_9bcf5eb9c288.slice/crio-5008083d08947ff65e2f7020748b6e83f622f263681e64470432e72dbea6018c WatchSource:0}: Error finding container 5008083d08947ff65e2f7020748b6e83f622f263681e64470432e72dbea6018c: Status 404 returned error can't find the container with id 5008083d08947ff65e2f7020748b6e83f622f263681e64470432e72dbea6018c Nov 01 05:03:35.114137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.114102 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsvch" event={"ID":"048500be-6716-42e5-93e3-e3b7e3507882","Type":"ContainerStarted","Data":"43c6be8b66347207c1bfe41a8039b8b17f2c9ee2ea1f1e6bef79fc8b08f279c5"} Nov 01 05:03:35.114635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.114164 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsvch" event={"ID":"048500be-6716-42e5-93e3-e3b7e3507882","Type":"ContainerStarted","Data":"09194adc59022521945aeca7616f1a3679841ddc628f6f9225e4e115225a567d"} Nov 01 05:03:35.116295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.116267 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" event={"ID":"d5ddfea2-3759-4ad2-9b50-48acaf309f1f","Type":"ContainerStarted","Data":"a64d30c78e882e832dec35614eb7d51a3272c6fb9ca36aec9725015c10353a68"} Nov 01 05:03:35.117375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.117355 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"5008083d08947ff65e2f7020748b6e83f622f263681e64470432e72dbea6018c"} Nov 01 05:03:35.237061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.236967 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zsvch" podStartSLOduration=2.751808074 podStartE2EDuration="4.236952847s" totalImagesPullingTime="1.401223242s" totalInitContainerRuntime="83.921531ms" isStatefulPod=true podCreationTimestamp="2026-11-01 05:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:35.236345386 +0000 UTC m=+30.376934305" watchObservedRunningTime="2026-11-01 05:03:35.236952847 +0000 UTC m=+30.377541762" Nov 01 05:03:35.355778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:35.355730 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rf9s2" podStartSLOduration=2.894556188 podStartE2EDuration="4.355714231s" totalImagesPullingTime="1.461158043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:35.355277643 +0000 UTC m=+30.495866580" watchObservedRunningTime="2026-11-01 05:03:35.355714231 +0000 UTC m=+30.496303148" Nov 01 05:03:36.599130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.599094 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vbms"] Nov 01 05:03:36.601136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.601104 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.654751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.654721 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 05:03:36.662326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.662305 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-275hk\"" Nov 01 05:03:36.667374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.667349 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vbms"] Nov 01 05:03:36.670492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.670467 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.670652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.670627 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvsq\" (UniqueName: \"kubernetes.io/projected/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-kube-api-access-qnvsq\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.772023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.771947 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvsq\" (UniqueName: \"kubernetes.io/projected/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-kube-api-access-qnvsq\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.772023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.771991 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.774584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.774561 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.836956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.836918 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvsq\" (UniqueName: \"kubernetes.io/projected/2278e4b7-d92f-491a-b98b-ad1203cd8b9c-kube-api-access-qnvsq\") pod \"monitoring-plugin-77f4f76594-2vbms\" (UID: \"2278e4b7-d92f-491a-b98b-ad1203cd8b9c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:36.911108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:36.911074 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:37.126520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:37.126492 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25" exitCode=0 Nov 01 05:03:37.126695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:37.126555 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25"} Nov 01 05:03:37.283125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:37.283098 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vbms"] Nov 01 05:03:37.320459 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:37.320421 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2278e4b7_d92f_491a_b98b_ad1203cd8b9c.slice/crio-a37860aef310b1b6389806dc1c84f70277eb49990473b34aeda6b139ed9046da WatchSource:0}: Error finding container a37860aef310b1b6389806dc1c84f70277eb49990473b34aeda6b139ed9046da: Status 404 returned error can't find the container with id a37860aef310b1b6389806dc1c84f70277eb49990473b34aeda6b139ed9046da Nov 01 05:03:38.131056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:38.131021 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" event={"ID":"2278e4b7-d92f-491a-b98b-ad1203cd8b9c","Type":"ContainerStarted","Data":"a37860aef310b1b6389806dc1c84f70277eb49990473b34aeda6b139ed9046da"} Nov 01 05:03:39.028226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:39.028197 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5fx5x" Nov 01 05:03:40.141777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.141744 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8"} Nov 01 05:03:40.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.141822 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f"} Nov 01 05:03:40.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.141851 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a"} Nov 01 05:03:40.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.141876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6"} Nov 01 05:03:40.142222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.141913 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c"} Nov 01 05:03:40.143210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.143190 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" event={"ID":"2278e4b7-d92f-491a-b98b-ad1203cd8b9c","Type":"ContainerStarted","Data":"d8a31bef646bfde43e6c77204246c9321f50819471a468e4ec99daa115d1c761"} Nov 01 05:03:40.143391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.143363 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:40.148457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.148437 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" Nov 01 05:03:40.262049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:40.261967 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vbms" podStartSLOduration=2.484257367 podStartE2EDuration="4.261953626s" totalImagesPullingTime="1.777696259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:40.26122955 +0000 UTC m=+35.401818470" watchObservedRunningTime="2026-11-01 05:03:40.261953626 +0000 UTC m=+35.402542545" Nov 01 05:03:41.057238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.057204 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:03:41.059726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.059709 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.116583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.116492 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 05:03:41.116583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.116521 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 05:03:41.116583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.116498 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 05:03:41.116583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.116558 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 05:03:41.150885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.150849 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerStarted","Data":"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56"} Nov 01 05:03:41.166911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.166884 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 05:03:41.167214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.167071 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 05:03:41.167214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.167077 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gxrph\"" Nov 01 05:03:41.169768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.168535 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:03:41.209791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.209760 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 05:03:41.214137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214114 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214145 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214168 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214185 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214243 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214336 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214396 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214414 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214434 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4bw\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214455 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214537 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214579 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214637 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214682 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214702 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214719 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214734 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214752 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214800 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.214880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.214827 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.218068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.218050 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 05:03:41.218109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.218062 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 05:03:41.218239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.218224 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 05:03:41.293144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293107 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 05:03:41.293144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293123 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 05:03:41.293390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293158 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 05:03:41.293390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293174 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 05:03:41.293390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293354 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 05:03:41.293504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.293455 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-790lpj2p1uts6\"" Nov 01 05:03:41.315638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315615 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315656 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315672 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315693 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315721 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315786 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315830 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315910 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.315993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.315951 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316040 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316091 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316126 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316165 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316194 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316230 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4bw\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316359 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316428 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316475 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316515 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316753 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316818 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.316910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316890 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.317018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.316968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.319209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.318228 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.319209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.318394 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.319209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.318868 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.320318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.320292 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.320557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.320533 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.321042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.320667 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.321042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.320899 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.321175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.321042 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.321972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.321946 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.322071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.322044 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.322367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.322335 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.322536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.322513 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.322748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.322723 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.322839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.322821 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.323415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.323398 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.376410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.376316 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.438736841 podStartE2EDuration="7.376301848s" totalImagesPullingTime="4.924932667s" totalInitContainerRuntime="12.63234ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:41.375920797 +0000 UTC m=+36.516509737" watchObservedRunningTime="2026-11-01 05:03:41.376301848 +0000 UTC m=+36.516890766" Nov 01 05:03:41.384277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.384230 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4bw\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\") pod \"prometheus-k8s-0\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:41.675691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:41.675608 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:42.168529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:42.168485 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:03:42.220044 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:03:42.220016 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0be27f_3535_49a5_9062_03bd7070004a.slice/crio-4ad912fe70c91a6943e78dc3f3045cd6f36286957e4b310d4c5cc6ad86680936 WatchSource:0}: Error finding container 4ad912fe70c91a6943e78dc3f3045cd6f36286957e4b310d4c5cc6ad86680936: Status 404 returned error can't find the container with id 4ad912fe70c91a6943e78dc3f3045cd6f36286957e4b310d4c5cc6ad86680936 Nov 01 05:03:43.160492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:43.160456 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" exitCode=0 Nov 01 05:03:43.160657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:43.160515 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} Nov 01 05:03:43.160657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:43.160587 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"4ad912fe70c91a6943e78dc3f3045cd6f36286957e4b310d4c5cc6ad86680936"} Nov 01 05:03:44.066976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:44.066944 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9b96bb796-dq65h" Nov 01 05:03:47.178946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:47.178915 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} Nov 01 05:03:47.179372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:47.178980 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} Nov 01 05:03:49.194938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:49.194902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} Nov 01 05:03:50.201947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:50.201899 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} Nov 01 05:03:50.202411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:50.201975 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} Nov 01 05:03:50.202411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:50.202383 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerStarted","Data":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} Nov 01 05:03:50.488889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:50.488786 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.674678533 podStartE2EDuration="10.488772089s" totalImagesPullingTime="5.798802828s" totalInitContainerRuntime="15.290728ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:50.436941464 +0000 UTC m=+45.577530399" watchObservedRunningTime="2026-11-01 05:03:50.488772089 +0000 UTC m=+45.629361008" Nov 01 05:03:51.675972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:03:51.675935 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:03:56.947590 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:03:56.947550 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:04:24.053843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:24.053795 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:24.072559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:24.072536 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:24.340673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:24.340646 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:26.947018 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:26.946981 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:04:37.824839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.824809 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:37.825309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825224 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="alertmanager" containerID="cri-o://1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c" gracePeriod=120 Nov 01 05:04:37.825388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825291 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-metric" containerID="cri-o://29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8" gracePeriod=120 Nov 01 05:04:37.825388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825326 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="prom-label-proxy" containerID="cri-o://5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56" gracePeriod=120 Nov 01 05:04:37.825388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825347 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="config-reloader" containerID="cri-o://400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6" gracePeriod=120 Nov 01 05:04:37.825530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825369 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy" containerID="cri-o://126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f" gracePeriod=120 Nov 01 05:04:37.825530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:37.825334 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-web" containerID="cri-o://59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a" gracePeriod=120 Nov 01 05:04:38.382272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382231 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56" exitCode=0 Nov 01 05:04:38.382272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382269 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f" exitCode=0 Nov 01 05:04:38.382272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382275 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a" exitCode=0 Nov 01 05:04:38.382272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382281 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6" exitCode=0 Nov 01 05:04:38.382272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382286 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c" exitCode=0 Nov 01 05:04:38.382630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382308 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56"} Nov 01 05:04:38.382630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382389 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f"} Nov 01 05:04:38.382630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382431 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a"} Nov 01 05:04:38.382630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382467 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6"} Nov 01 05:04:38.382630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:38.382507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c"} Nov 01 05:04:39.078863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.078836 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.197385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197304 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197340 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197376 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197397 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197425 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197454 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197480 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197514 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197552 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197590 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197618 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.197657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197641 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.198096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197669 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\") pod \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\" (UID: \"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288\") " Nov 01 05:04:39.198096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197715 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:04:39.198096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.197981 4425 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-main-db\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.198096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.198051 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:39.198326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.198297 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:39.200431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.200404 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:04:39.201111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.201068 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:04:39.201459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.201413 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.201712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.201679 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.201879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.201851 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "kube-api-access-l7prx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:04:39.201978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.201933 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.202472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.202447 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.202717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.202693 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.205492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.205397 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.211842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.211819 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config" pod "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" (UID: "2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:39.299003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.298966 4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.298997 4425 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-web-config\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299011 4425 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299021 4425 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-tls-assets\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299029 4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-main-tls\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299038 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7prx\" (UniqueName: \"kubernetes.io/projected/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-kube-api-access-l7prx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299046 4425 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-metrics-client-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299056 4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299064 4425 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-cluster-tls-config\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299073 4425 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-out\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299081 4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.299224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.299089 4425 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:39.390541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.390507 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerID="29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8" exitCode=0 Nov 01 05:04:39.390732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.390580 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8"} Nov 01 05:04:39.390732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.390619 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.390732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.390657 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288","Type":"ContainerDied","Data":"5008083d08947ff65e2f7020748b6e83f622f263681e64470432e72dbea6018c"} Nov 01 05:04:39.390732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.390716 4425 scope.go:122] "RemoveContainer" containerID="5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56" Nov 01 05:04:39.400745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.400728 4425 scope.go:122] "RemoveContainer" containerID="29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8" Nov 01 05:04:39.409834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.409814 4425 scope.go:122] "RemoveContainer" containerID="126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f" Nov 01 05:04:39.417293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.417239 4425 scope.go:122] "RemoveContainer" containerID="59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a" Nov 01 05:04:39.426041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.426019 4425 scope.go:122] "RemoveContainer" containerID="400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6" Nov 01 05:04:39.433602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.433579 4425 scope.go:122] "RemoveContainer" containerID="1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c" Nov 01 05:04:39.441005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.440987 4425 scope.go:122] "RemoveContainer" containerID="e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25" Nov 01 05:04:39.448243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.448226 4425 scope.go:122] "RemoveContainer" containerID="5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56" Nov 01 05:04:39.448550 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.448529 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56\": container with ID starting with 5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56 not found: ID does not exist" containerID="5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56" Nov 01 05:04:39.448603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.448561 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56"} err="failed to get container status \"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56\": rpc error: code = NotFound desc = could not find container \"5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56\": container with ID starting with 5f42da2d452901ac8a299729fd5c79363c6097c18d4a1b560627f1876bd55b56 not found: ID does not exist" Nov 01 05:04:39.448603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.448594 4425 scope.go:122] "RemoveContainer" containerID="29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8" Nov 01 05:04:39.448824 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.448809 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8\": container with ID starting with 29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8 not found: ID does not exist" containerID="29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8" Nov 01 05:04:39.448866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.448828 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8"} err="failed to get container status \"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8\": rpc error: code = NotFound desc = could not find container \"29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8\": container with ID starting with 29ce6323433063c45bdf20372f1964a519cd8d51e901485ff9a55693d2962fe8 not found: ID does not exist" Nov 01 05:04:39.448866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.448840 4425 scope.go:122] "RemoveContainer" containerID="126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f" Nov 01 05:04:39.449049 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.449032 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f\": container with ID starting with 126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f not found: ID does not exist" containerID="126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f" Nov 01 05:04:39.449114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449057 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f"} err="failed to get container status \"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f\": rpc error: code = NotFound desc = could not find container \"126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f\": container with ID starting with 126ec8af589a4f613acafb4ee6f80a979f9932bcc935c74e035ccae0441b890f not found: ID does not exist" Nov 01 05:04:39.449114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449078 4425 scope.go:122] "RemoveContainer" containerID="59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a" Nov 01 05:04:39.449343 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.449328 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a\": container with ID starting with 59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a not found: ID does not exist" containerID="59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a" Nov 01 05:04:39.449385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449346 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a"} err="failed to get container status \"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a\": rpc error: code = NotFound desc = could not find container \"59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a\": container with ID starting with 59328c392293b847523122756d04900e1e3a84a813d624209415cc131c5d4f0a not found: ID does not exist" Nov 01 05:04:39.449385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449360 4425 scope.go:122] "RemoveContainer" containerID="400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6" Nov 01 05:04:39.449579 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.449562 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6\": container with ID starting with 400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6 not found: ID does not exist" containerID="400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6" Nov 01 05:04:39.449642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449586 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6"} err="failed to get container status \"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6\": rpc error: code = NotFound desc = could not find container \"400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6\": container with ID starting with 400ed260920428a7a7274bd2a47c737dfbd72fb5d48ee22c389052c5e73179c6 not found: ID does not exist" Nov 01 05:04:39.449642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449609 4425 scope.go:122] "RemoveContainer" containerID="1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c" Nov 01 05:04:39.449878 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.449861 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c\": container with ID starting with 1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c not found: ID does not exist" containerID="1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c" Nov 01 05:04:39.449911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449883 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c"} err="failed to get container status \"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c\": rpc error: code = NotFound desc = could not find container \"1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c\": container with ID starting with 1b9baedd8b3dce781d7821c0a016b4b11164fb1cb66f5f31b7d3c33db54d3e5c not found: ID does not exist" Nov 01 05:04:39.449911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.449896 4425 scope.go:122] "RemoveContainer" containerID="e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25" Nov 01 05:04:39.450185 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:39.450164 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25\": container with ID starting with e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25 not found: ID does not exist" containerID="e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25" Nov 01 05:04:39.450301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.450190 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25"} err="failed to get container status \"e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25\": rpc error: code = NotFound desc = could not find container \"e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25\": container with ID starting with e419549b30d3587043fb1863ec1e918d6641fe7df0979f0f72adf3de80697b25 not found: ID does not exist" Nov 01 05:04:39.560771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.560735 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:39.564774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.564746 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:39.611405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611379 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:39.611639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611627 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-web" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611640 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-web" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611653 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="alertmanager" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611658 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="alertmanager" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611664 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611669 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611678 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="prom-label-proxy" Nov 01 05:04:39.611680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611682 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="prom-label-proxy" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611690 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="init-config-reloader" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611694 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="init-config-reloader" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611702 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="config-reloader" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611708 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="config-reloader" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611721 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-metric" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611726 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-metric" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611772 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-web" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611778 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="alertmanager" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611785 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611791 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="prom-label-proxy" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611796 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="config-reloader" Nov 01 05:04:39.611894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.611802 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" containerName="kube-rbac-proxy-metric" Nov 01 05:04:39.614989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.614973 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.668878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.668846 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 05:04:39.668878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.668874 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 05:04:39.669087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.668874 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.669957 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.670026 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.670175 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.670384 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8mzd7\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.670523 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 05:04:39.670845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.670802 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 05:04:39.673545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.671969 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 05:04:39.702118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702031 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702087 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702107 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702205 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702289 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702326 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702353 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702421 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702447 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702465 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702527 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rp5s\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-kube-api-access-6rp5s\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702549 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.702627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.702571 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.720701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.720673 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:39.803191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803153 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803201 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803219 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803240 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803297 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803315 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803342 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803810 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803865 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.803947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803895 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.804051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803951 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rp5s\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-kube-api-access-6rp5s\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.804051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.803992 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.804051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.804019 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.804210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.804077 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.804911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.804883 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.805119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.805094 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.806779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.806751 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807025 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.806997 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807030 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807033 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807033 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807282 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807417 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.807811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.807795 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.808724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.808703 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.866191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.866167 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rp5s\" (UniqueName: \"kubernetes.io/projected/b5aeb4ea-7f78-4711-be7b-e0b736015b9a-kube-api-access-6rp5s\") pod \"alertmanager-main-0\" (UID: \"b5aeb4ea-7f78-4711-be7b-e0b736015b9a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:39.930620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:39.930585 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 05:04:40.360542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:40.360513 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 05:04:40.363156 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:04:40.363127 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aeb4ea_7f78_4711_be7b_e0b736015b9a.slice/crio-2bd101b543f8ca1b781bc8eba5a659821d6d143673d78ed6e59779c962f06d3e WatchSource:0}: Error finding container 2bd101b543f8ca1b781bc8eba5a659821d6d143673d78ed6e59779c962f06d3e: Status 404 returned error can't find the container with id 2bd101b543f8ca1b781bc8eba5a659821d6d143673d78ed6e59779c962f06d3e Nov 01 05:04:40.395928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:40.395904 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"2bd101b543f8ca1b781bc8eba5a659821d6d143673d78ed6e59779c962f06d3e"} Nov 01 05:04:40.970323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:40.970288 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288" path="/var/lib/kubelet/pods/2b649fc4-8d29-4ebd-b7d8-9bcf5eb9c288/volumes" Nov 01 05:04:41.401113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:41.401079 4425 generic.go:356] "Generic (PLEG): container finished" podID="b5aeb4ea-7f78-4711-be7b-e0b736015b9a" containerID="291fc48c6de6e2311d76c6e8902e41381c07721f3466b84bbad4a00079ee5e95" exitCode=0 Nov 01 05:04:41.401556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:41.401123 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerDied","Data":"291fc48c6de6e2311d76c6e8902e41381c07721f3466b84bbad4a00079ee5e95"} Nov 01 05:04:42.407748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407714 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"3648aa3b09353aaaac5afe23bfb85c3129a05eeaf2ee8d25b7a6f729a9f43dde"} Nov 01 05:04:42.408174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407806 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"ddf84b348e84ebaa2a774ac53c4c017a9c16c07610954978e41b11145938e9e8"} Nov 01 05:04:42.408174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407833 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"c07ec326417e0f5d0dd131412426740d0ee799334cbe3c9980c575b3a89b948f"} Nov 01 05:04:42.408174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407862 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"790fe465ba79118b3318d9dce26fc5d57b023d32e4f2557a442d673e26747b43"} Nov 01 05:04:42.408174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407884 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"ad7b68b870109b525bb1f83d8eed8a802776f86e8406b41d26c494e95001e24e"} Nov 01 05:04:42.408174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.407908 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5aeb4ea-7f78-4711-be7b-e0b736015b9a","Type":"ContainerStarted","Data":"82a4ad5c6c500d416b6301c8b37268bb6bff3ab52700c8a3957458e8dd0299b1"} Nov 01 05:04:42.541802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:42.541748 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.528407838 podStartE2EDuration="3.54173445s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.326612ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:04:42.53886922 +0000 UTC m=+97.679458161" watchObservedRunningTime="2026-11-01 05:04:42.54173445 +0000 UTC m=+97.682323409" Nov 01 05:04:43.517939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.517896 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518304 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="prometheus" containerID="cri-o://0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" gracePeriod=600 Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518325 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-web" containerID="cri-o://dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" gracePeriod=600 Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518324 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy" containerID="cri-o://4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" gracePeriod=600 Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518357 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="config-reloader" containerID="cri-o://ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" gracePeriod=600 Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518407 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" gracePeriod=600 Nov 01 05:04:43.518469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.518325 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="thanos-sidecar" containerID="cri-o://c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" gracePeriod=600 Nov 01 05:04:43.769351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.769287 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:43.944473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944442 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944492 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944530 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944690 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944748 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944784 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944810 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944842 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944870 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944898 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.944978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.944965 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945023 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945058 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945105 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945137 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945219 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945407 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945475 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945512 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945551 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\") pod \"0f0be27f-3535-49a5-9062-03bd7070004a\" (UID: \"0f0be27f-3535-49a5-9062-03bd7070004a\") " Nov 01 05:04:43.945637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945562 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.945950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945824 4425 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:43.945950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.945929 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.946112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.946087 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:04:43.946192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.946170 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.946840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.946469 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.947074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.947043 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.947414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.947378 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.948149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.948118 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:04:43.948239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.948192 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.948650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.948623 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:04:43.950018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.949747 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.950018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.949856 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.950018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.949921 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:04:43.950222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.950136 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "kube-api-access-pn4bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:04:43.950222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.950194 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.950687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.950661 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.951288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.951241 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.951497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.951478 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.951707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.951676 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:43.960104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:43.960082 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config" pod "0f0be27f-3535-49a5-9062-03bd7070004a" (UID: "0f0be27f-3535-49a5-9062-03bd7070004a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046532 4425 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-grpc-tls\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046558 4425 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-web-config\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046569 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4bw\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-kube-api-access-pn4bw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046578 4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046586 4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046595 4425 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f0be27f-3535-49a5-9062-03bd7070004a-tls-assets\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046603 4425 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-config\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046611 4425 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046642 4425 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046651 4425 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-config-out\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046660 4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046670 4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-db\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046678 4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046689 4425 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-kube-rbac-proxy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046698 4425 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-metrics-client-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046707 4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046717 4425 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-configmap-metrics-client-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046725 4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f0be27f-3535-49a5-9062-03bd7070004a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.046899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.046733 4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f0be27f-3535-49a5-9062-03bd7070004a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:04:44.422907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422869 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" exitCode=0 Nov 01 05:04:44.422907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422901 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" exitCode=0 Nov 01 05:04:44.422907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422914 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" exitCode=0 Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422925 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" exitCode=0 Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422941 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" exitCode=0 Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422953 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f0be27f-3535-49a5-9062-03bd7070004a" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" exitCode=0 Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.422993 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423051 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423115 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} Nov 01 05:04:44.423203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423173 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} Nov 01 05:04:44.423540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423230 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} Nov 01 05:04:44.423540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423337 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} Nov 01 05:04:44.423540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423391 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} Nov 01 05:04:44.423540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423445 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f0be27f-3535-49a5-9062-03bd7070004a","Type":"ContainerDied","Data":"4ad912fe70c91a6943e78dc3f3045cd6f36286957e4b310d4c5cc6ad86680936"} Nov 01 05:04:44.423540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.423502 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.433148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.433124 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.443394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.443376 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.452804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.452785 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.462973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.461450 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.471739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.470130 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.479097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.479069 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.488663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.488638 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.489015 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.488993 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.489099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489033 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.489099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489061 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.489365 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.489346 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.489447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489375 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.489447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489397 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.489676 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.489659 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.489739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489683 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.489739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489702 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.489986 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.489957 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.490037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.489992 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.490037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.490012 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.490443 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.490327 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.490443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.490358 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.490443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.490381 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.490715 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.490696 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.490788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.490722 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.490788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.490741 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.491011 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:44.490992 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.491074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491019 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.491074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491037 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.491436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491347 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.491436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491371 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.491858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491771 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.491858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.491793 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.492206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492183 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.492206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492204 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.492503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492483 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.492579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492505 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.492741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492723 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.492822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492741 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.493000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.492980 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.493051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493007 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.493343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493323 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.493403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493344 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.493601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493582 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.493673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493603 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.493817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493799 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.493886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.493819 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.494051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494033 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.494091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494052 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.494290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494271 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.494372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494290 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.494499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494477 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.494537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494500 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.494708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494692 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.494770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494710 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.494942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494921 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.495007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.494946 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.495180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495163 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.495239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495181 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.495420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495402 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.495468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495420 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.495640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495612 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.495640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495630 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.495865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495845 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.495865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.495866 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.496088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496073 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.496131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496088 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.496306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496291 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.496355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496307 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.496496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496477 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.496537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496497 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.496680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496662 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.496719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496680 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.496911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496896 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.496955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.496912 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.497163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497125 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.497163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497155 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.497479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497447 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.497479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497470 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.497812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497790 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.497891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.497812 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.498089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498070 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.498156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498090 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498391 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498467 4425 scope.go:122] "RemoveContainer" containerID="1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498758 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946"} err="failed to get container status \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": rpc error: code = NotFound desc = could not find container \"1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946\": container with ID starting with 1aad720dee0be9e47a11bbe8f5ddee6618829d8a819b7189a26c5a0eb0e7d946 not found: ID does not exist" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.498783 4425 scope.go:122] "RemoveContainer" containerID="4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499086 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca"} err="failed to get container status \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": rpc error: code = NotFound desc = could not find container \"4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca\": container with ID starting with 4497b49bb97a99505468ea43c28c1daa513753438edd357aa7eceb9e6489f3ca not found: ID does not exist" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499108 4425 scope.go:122] "RemoveContainer" containerID="dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499432 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341"} err="failed to get container status \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": rpc error: code = NotFound desc = could not find container \"dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341\": container with ID starting with dc26881171f7d81acc89e33f876001a0424dcf209bd7a3bd8e7d55a8e87f1341 not found: ID does not exist" Nov 01 05:04:44.499473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499454 4425 scope.go:122] "RemoveContainer" containerID="c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6" Nov 01 05:04:44.499879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499706 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6"} err="failed to get container status \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": rpc error: code = NotFound desc = could not find container \"c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6\": container with ID starting with c344197586fc2078e3c025dd2b514fb417fe708911ba875d2d1fdc32ab6821c6 not found: ID does not exist" Nov 01 05:04:44.499879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499722 4425 scope.go:122] "RemoveContainer" containerID="ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6" Nov 01 05:04:44.499970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499950 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6"} err="failed to get container status \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": rpc error: code = NotFound desc = could not find container \"ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6\": container with ID starting with ac25a6372eea9676ce2e8d12225578602e8e6e19c140c67d8ff4a806ecd8d1d6 not found: ID does not exist" Nov 01 05:04:44.499970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.499965 4425 scope.go:122] "RemoveContainer" containerID="0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6" Nov 01 05:04:44.500201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.500181 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6"} err="failed to get container status \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": rpc error: code = NotFound desc = could not find container \"0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6\": container with ID starting with 0a6fab06ffecb7dece71ff04eece0b3b2f2d3e904d1b85d7b360203b2f363fc6 not found: ID does not exist" Nov 01 05:04:44.500280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.500202 4425 scope.go:122] "RemoveContainer" containerID="97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c" Nov 01 05:04:44.500490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.500473 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c"} err="failed to get container status \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": rpc error: code = NotFound desc = could not find container \"97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c\": container with ID starting with 97e0cf2c1972af53035d6a7914f3ae05a59403c1fb48bd076d09c53e5a75a54c not found: ID does not exist" Nov 01 05:04:44.645761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.645723 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:44.696167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.696093 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:44.696856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.696839 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:44.697117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697103 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-thanos" Nov 01 05:04:44.697155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697120 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-thanos" Nov 01 05:04:44.697155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697134 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="config-reloader" Nov 01 05:04:44.697155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697139 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="config-reloader" Nov 01 05:04:44.697155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697152 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="init-config-reloader" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697156 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="init-config-reloader" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697170 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697177 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697194 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="prometheus" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697199 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="prometheus" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697206 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-web" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697211 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-web" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697219 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="thanos-sidecar" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697223 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="thanos-sidecar" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697286 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="config-reloader" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697297 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy" Nov 01 05:04:44.697298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697303 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="prometheus" Nov 01 05:04:44.697912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697308 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-web" Nov 01 05:04:44.697912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697314 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="thanos-sidecar" Nov 01 05:04:44.697912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.697320 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" containerName="kube-rbac-proxy-thanos" Nov 01 05:04:44.700637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.700621 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.754589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754554 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 05:04:44.754757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754667 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 05:04:44.754757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754676 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 05:04:44.754757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754666 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 05:04:44.754757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754725 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 05:04:44.754991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754898 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 05:04:44.754991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.754911 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 05:04:44.755088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755035 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 05:04:44.755088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755040 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 05:04:44.755445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755430 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gxrph\"" Nov 01 05:04:44.755544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755450 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 05:04:44.755544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755469 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 05:04:44.755544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755493 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 05:04:44.755674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755561 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 05:04:44.755674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.755573 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-790lpj2p1uts6\"" Nov 01 05:04:44.806687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.806662 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 05:04:44.806687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.806678 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 05:04:44.816999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.816975 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:44.851768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851739 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.851768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851767 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851787 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851804 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851868 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.851940 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852023 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-web-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852055 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-config-out\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852094 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7lq\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-kube-api-access-8d7lq\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852155 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852194 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852280 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852340 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852365 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852382 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852455 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852486 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852506 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.852564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.852521 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953399 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953445 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953467 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953493 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953522 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953551 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953585 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-web-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.953829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.953629 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-config-out\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954553 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954617 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7lq\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-kube-api-access-8d7lq\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954668 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954701 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954728 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954769 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954800 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954830 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.954896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954859 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954900 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954930 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954954 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.954976 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.955098 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.955545 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.955850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.955654 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.956986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.956627 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-config-out\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.956986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.956943 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-web-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.957136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.957089 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.957330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.957303 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.957723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.957412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.957723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.957661 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.958121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.958095 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.958667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.958639 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.959001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.958970 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.959089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.959051 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f00763-56d4-448a-92c8-278e091e1206-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.959621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.959602 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.959724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.959703 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.959995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.959969 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.960104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.960083 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.960223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.960209 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/92f00763-56d4-448a-92c8-278e091e1206-config\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:44.970780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:44.970759 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0be27f-3535-49a5-9062-03bd7070004a" path="/var/lib/kubelet/pods/0f0be27f-3535-49a5-9062-03bd7070004a/volumes" Nov 01 05:04:45.018651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:45.018621 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7lq\" (UniqueName: \"kubernetes.io/projected/92f00763-56d4-448a-92c8-278e091e1206-kube-api-access-8d7lq\") pod \"prometheus-k8s-0\" (UID: \"92f00763-56d4-448a-92c8-278e091e1206\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:45.317271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:45.317225 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:45.800660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:45.800633 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 05:04:45.802619 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:04:45.802594 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f00763_56d4_448a_92c8_278e091e1206.slice/crio-25603406a30f3b52c56cbf8f92dd6a5a527ae05cf3e82d86add1c6e13beb7618 WatchSource:0}: Error finding container 25603406a30f3b52c56cbf8f92dd6a5a527ae05cf3e82d86add1c6e13beb7618: Status 404 returned error can't find the container with id 25603406a30f3b52c56cbf8f92dd6a5a527ae05cf3e82d86add1c6e13beb7618 Nov 01 05:04:46.432122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:46.432086 4425 generic.go:356] "Generic (PLEG): container finished" podID="92f00763-56d4-448a-92c8-278e091e1206" containerID="de6947baf6360e294e588986e01eda481c17ad5aca02cd12cafab561353d845a" exitCode=0 Nov 01 05:04:46.432283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:46.432159 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerDied","Data":"de6947baf6360e294e588986e01eda481c17ad5aca02cd12cafab561353d845a"} Nov 01 05:04:46.432283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:46.432225 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"25603406a30f3b52c56cbf8f92dd6a5a527ae05cf3e82d86add1c6e13beb7618"} Nov 01 05:04:47.439868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439833 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"c9c1b99e001c5898d37e2f7605260e8bc4b31fd9fee4bf1d9d1ea241a5be2e13"} Nov 01 05:04:47.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439896 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"0671269a7ee443d978397a8e8f9192beab89033f056d1eac4af4b2eb81469423"} Nov 01 05:04:47.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439923 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"bcadde2ffefc773f3839990d41b09998f1b23afd44ce87c4f8bf66b0fdd2818d"} Nov 01 05:04:47.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439949 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"7ab3ee1ae6e34ba949530145067f151fab669fbc1102b3556f2130d9c45f436a"} Nov 01 05:04:47.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439973 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"7f405941378abd3fe0552646588fd54962c011454d6b10ca03abb4b5db0630df"} Nov 01 05:04:47.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.439994 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92f00763-56d4-448a-92c8-278e091e1206","Type":"ContainerStarted","Data":"47345a89cbb68461dab27d9349b19e3d1c0bb45c8d48f5886625ff3f57b34232"} Nov 01 05:04:47.574496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:47.574447 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.557420519 podStartE2EDuration="3.574429573s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.009054ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:04:47.572581744 +0000 UTC m=+102.713170663" watchObservedRunningTime="2026-11-01 05:04:47.574429573 +0000 UTC m=+102.715018890" Nov 01 05:04:50.318135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:04:50.318098 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:04:56.947376 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:04:56.947336 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:05:26.947468 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:05:26.947385 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:05:45.317689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:05:45.317653 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:05:45.334159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:05:45.334133 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:05:45.659150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:05:45.659058 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 05:05:56.946216 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:05:56.946178 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:06:26.946229 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:06:26.946189 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:06:56.945903 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:06:56.945825 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:07:26.946793 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:07:26.946759 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:07:56.946272 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:07:56.946214 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:08:26.945917 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:08:26.945719 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:08:35.081494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.081460 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd"] Nov 01 05:08:35.084719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.084703 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190611 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190654 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190746 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190782 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190839 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.190916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.190893 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twlbn\" (UniqueName: \"kubernetes.io/projected/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-kube-api-access-twlbn\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.212661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212627 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 05:08:35.212661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212649 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 05:08:35.212865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212692 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 05:08:35.212865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212712 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 05:08:35.212865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212747 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 05:08:35.212865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.212801 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-75h7v\"" Nov 01 05:08:35.233626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.233600 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd"] Nov 01 05:08:35.291353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291320 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twlbn\" (UniqueName: \"kubernetes.io/projected/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-kube-api-access-twlbn\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291354 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291386 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291424 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291442 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291471 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.291887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.291862 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.294085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.294062 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.294248 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.294225 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.294325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.294238 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.294325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.294295 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.353512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.353452 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twlbn\" (UniqueName: \"kubernetes.io/projected/cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b-kube-api-access-twlbn\") pod \"opendatahub-operator-controller-manager-85d75859fc-5shzd\" (UID: \"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.400455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.400423 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:35.893490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.893466 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd"] Nov 01 05:08:35.895168 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:08:35.895141 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9d6c8e_cccc_4c49_9a8f_4c2c83e2cc0b.slice/crio-f64778eb68a9b2bebf7a46ef98736ec77f88c4a1d46c115752750c0011894b79 WatchSource:0}: Error finding container f64778eb68a9b2bebf7a46ef98736ec77f88c4a1d46c115752750c0011894b79: Status 404 returned error can't find the container with id f64778eb68a9b2bebf7a46ef98736ec77f88c4a1d46c115752750c0011894b79 Nov 01 05:08:35.896843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:35.896803 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:08:36.238264 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:36.238171 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" event={"ID":"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b","Type":"ContainerStarted","Data":"f64778eb68a9b2bebf7a46ef98736ec77f88c4a1d46c115752750c0011894b79"} Nov 01 05:08:40.258516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:40.258482 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" event={"ID":"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b","Type":"ContainerStarted","Data":"99a9f2fbf68324e12b82a0c6987873e1c74ad9221e11f30021d6d671a9e33c70"} Nov 01 05:08:41.264059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:41.264024 4425 generic.go:356] "Generic (PLEG): container finished" podID="cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b" containerID="99a9f2fbf68324e12b82a0c6987873e1c74ad9221e11f30021d6d671a9e33c70" exitCode=0 Nov 01 05:08:41.264523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:41.264086 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" event={"ID":"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b","Type":"ContainerDied","Data":"99a9f2fbf68324e12b82a0c6987873e1c74ad9221e11f30021d6d671a9e33c70"} Nov 01 05:08:42.269237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:42.269197 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" event={"ID":"cc9d6c8e-cccc-4c49-9a8f-4c2c83e2cc0b","Type":"ContainerStarted","Data":"9dd01a5b041b77715500c14ca4cb1dfd512b360e2dd74d64d1a37bb53cf02f78"} Nov 01 05:08:42.269669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:42.269323 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:42.519469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:42.519365 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" podStartSLOduration=2.867401897 podStartE2EDuration="8.519347871s" totalImagesPullingTime="5.158455424s" totalInitContainerRuntime="493.49055ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:08:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:08:42.467884885 +0000 UTC m=+337.608473841" watchObservedRunningTime="2026-11-01 05:08:42.519347871 +0000 UTC m=+337.659936790" Nov 01 05:08:53.275689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:53.275647 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-5shzd" Nov 01 05:08:55.097383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.097349 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-994b7"] Nov 01 05:08:55.102564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.102539 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.158798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.158765 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8wjdv\"" Nov 01 05:08:55.162408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.162376 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 05:08:55.163669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.163642 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcbr\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-kube-api-access-skcbr\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.163801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.163720 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.209744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.209713 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 05:08:55.211162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.211138 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-994b7"] Nov 01 05:08:55.264484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.264452 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.264642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.264508 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcbr\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-kube-api-access-skcbr\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.329194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.329158 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.330357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.330336 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcbr\" (UniqueName: \"kubernetes.io/projected/d6a2c42a-148c-4723-a5e9-4bdb8a2da212-kube-api-access-skcbr\") pod \"cert-manager-webhook-5dbbbf75cc-994b7\" (UID: \"d6a2c42a-148c-4723-a5e9-4bdb8a2da212\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.425552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.425473 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:08:55.803874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:55.803849 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-994b7"] Nov 01 05:08:55.806673 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:08:55.806645 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a2c42a_148c_4723_a5e9_4bdb8a2da212.slice/crio-256320d50ab85e1cd3333d6b9c1938392615fef6c0830fc1eb0a1d9c0634ad36 WatchSource:0}: Error finding container 256320d50ab85e1cd3333d6b9c1938392615fef6c0830fc1eb0a1d9c0634ad36: Status 404 returned error can't find the container with id 256320d50ab85e1cd3333d6b9c1938392615fef6c0830fc1eb0a1d9c0634ad36 Nov 01 05:08:56.321581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:56.321548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" event={"ID":"d6a2c42a-148c-4723-a5e9-4bdb8a2da212","Type":"ContainerStarted","Data":"256320d50ab85e1cd3333d6b9c1938392615fef6c0830fc1eb0a1d9c0634ad36"} Nov 01 05:08:56.946751 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:08:56.946708 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:08:57.175596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.175561 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7"] Nov 01 05:08:57.180279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.180241 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.237165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.237079 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7"] Nov 01 05:08:57.281164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.281118 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnjg\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-kube-api-access-srnjg\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.281164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.281173 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.286353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.286335 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f7k77\"" Nov 01 05:08:57.381913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.381857 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnjg\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-kube-api-access-srnjg\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.381913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.381919 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.444055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.444019 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.444209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.444086 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnjg\" (UniqueName: \"kubernetes.io/projected/cfdd9691-4956-4df0-a6f9-8889d74d090c-kube-api-access-srnjg\") pod \"cert-manager-cainjector-5c57bbd4fc-kc9p7\" (UID: \"cfdd9691-4956-4df0-a6f9-8889d74d090c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.491033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.490945 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" Nov 01 05:08:57.859852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:57.859813 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7"] Nov 01 05:08:57.863502 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:08:57.863470 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfdd9691_4956_4df0_a6f9_8889d74d090c.slice/crio-34ebe164f2fc1ee71297ea2fbd834623aa08af1413b404f3c66cb294a487d606 WatchSource:0}: Error finding container 34ebe164f2fc1ee71297ea2fbd834623aa08af1413b404f3c66cb294a487d606: Status 404 returned error can't find the container with id 34ebe164f2fc1ee71297ea2fbd834623aa08af1413b404f3c66cb294a487d606 Nov 01 05:08:58.331808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:08:58.331771 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" event={"ID":"cfdd9691-4956-4df0-a6f9-8889d74d090c","Type":"ContainerStarted","Data":"34ebe164f2fc1ee71297ea2fbd834623aa08af1413b404f3c66cb294a487d606"} Nov 01 05:09:01.353430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:01.353394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" event={"ID":"d6a2c42a-148c-4723-a5e9-4bdb8a2da212","Type":"ContainerStarted","Data":"51358a3ed2edc4fdf6136a4c5ff764e0a5f658f4d424117a0aea9c1f6de7c86c"} Nov 01 05:09:01.353882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:01.353495 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:09:01.475383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:01.475278 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" podStartSLOduration=1.068994288 podStartE2EDuration="6.475246834s" totalImagesPullingTime="5.406252546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:01.474460997 +0000 UTC m=+356.615049918" watchObservedRunningTime="2026-11-01 05:09:01.475246834 +0000 UTC m=+356.615835753" Nov 01 05:09:02.358507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:02.358463 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" event={"ID":"cfdd9691-4956-4df0-a6f9-8889d74d090c","Type":"ContainerStarted","Data":"50335ff1009b5309900a5ea9b2b30a994fcd119bfb733f3beca208d163d32423"} Nov 01 05:09:02.481451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:02.481387 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-kc9p7" podStartSLOduration=1.74302196 podStartE2EDuration="5.481372378s" totalImagesPullingTime="3.738350418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:02.480174867 +0000 UTC m=+357.620763788" watchObservedRunningTime="2026-11-01 05:09:02.481372378 +0000 UTC m=+357.621961297" Nov 01 05:09:07.361271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:07.361225 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-994b7" Nov 01 05:09:08.024501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.024460 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z"] Nov 01 05:09:08.029128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.029102 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.085926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.085894 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 05:09:08.086271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.085965 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 05:09:08.137618 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.137573 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z"] Nov 01 05:09:08.170160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.170132 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 05:09:08.170425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.170399 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 05:09:08.170551 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.170489 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ql6nz\"" Nov 01 05:09:08.173759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.173735 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.173839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.173790 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tmp\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.173875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.173851 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zzf\" (UniqueName: \"kubernetes.io/projected/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-kube-api-access-b6zzf\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.274765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.274676 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.274765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.274746 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tmp\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.274957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.274777 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zzf\" (UniqueName: \"kubernetes.io/projected/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-kube-api-access-b6zzf\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.277317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.277289 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tmp\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.277445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.277424 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.338553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.338518 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zzf\" (UniqueName: \"kubernetes.io/projected/dd2e9d9d-ffe8-44c0-8a9f-726020910c14-kube-api-access-b6zzf\") pod \"kube-auth-proxy-5f68b7bd84-nfg9z\" (UID: \"dd2e9d9d-ffe8-44c0-8a9f-726020910c14\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.340749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.340732 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" Nov 01 05:09:08.716501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:08.716479 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z"] Nov 01 05:09:08.718971 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:09:08.718943 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2e9d9d_ffe8_44c0_8a9f_726020910c14.slice/crio-32d997e67db0cf82e2225dccedc3d1bd8672b173a47c6046c8a16ba04c891cc4 WatchSource:0}: Error finding container 32d997e67db0cf82e2225dccedc3d1bd8672b173a47c6046c8a16ba04c891cc4: Status 404 returned error can't find the container with id 32d997e67db0cf82e2225dccedc3d1bd8672b173a47c6046c8a16ba04c891cc4 Nov 01 05:09:09.389474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:09.389443 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" event={"ID":"dd2e9d9d-ffe8-44c0-8a9f-726020910c14","Type":"ContainerStarted","Data":"32d997e67db0cf82e2225dccedc3d1bd8672b173a47c6046c8a16ba04c891cc4"} Nov 01 05:09:12.404933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:12.404893 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" event={"ID":"dd2e9d9d-ffe8-44c0-8a9f-726020910c14","Type":"ContainerStarted","Data":"ed45ad97db6d99ad999a9e9af8bca789e4739aa205a0c564513b8a239fe11075"} Nov 01 05:09:12.523215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:12.523162 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-nfg9z" podStartSLOduration=2.759261378 podStartE2EDuration="5.523147255s" totalImagesPullingTime="2.763885877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:12.522927819 +0000 UTC m=+367.663516739" watchObservedRunningTime="2026-11-01 05:09:12.523147255 +0000 UTC m=+367.663736174" Nov 01 05:09:15.214713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.214681 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-fztvp"] Nov 01 05:09:15.217983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.217964 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.324977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.324946 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-blh5c\"" Nov 01 05:09:15.326327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.326295 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fztvp"] Nov 01 05:09:15.347480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.347450 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-bound-sa-token\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.347636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.347570 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69wq\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-kube-api-access-x69wq\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.448630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.448580 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-bound-sa-token\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.448814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.448667 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69wq\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-kube-api-access-x69wq\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.510960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.510891 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69wq\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-kube-api-access-x69wq\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.510960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.510899 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98a3da61-90d5-4b7d-842c-683572d6bfbf-bound-sa-token\") pod \"cert-manager-597b95cb84-fztvp\" (UID: \"98a3da61-90d5-4b7d-842c-683572d6bfbf\") " pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.529074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.529036 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fztvp" Nov 01 05:09:15.901871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:15.901846 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fztvp"] Nov 01 05:09:15.903609 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:09:15.903580 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a3da61_90d5_4b7d_842c_683572d6bfbf.slice/crio-e24332c933091303d03c7234c6403d377854af1e62b6f430585c2de333543d09 WatchSource:0}: Error finding container e24332c933091303d03c7234c6403d377854af1e62b6f430585c2de333543d09: Status 404 returned error can't find the container with id e24332c933091303d03c7234c6403d377854af1e62b6f430585c2de333543d09 Nov 01 05:09:16.424418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:16.424383 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fztvp" event={"ID":"98a3da61-90d5-4b7d-842c-683572d6bfbf","Type":"ContainerStarted","Data":"0bad1a747505d4a8fade3dab0b95d0fe29768e8b3420120368864b1457a90d41"} Nov 01 05:09:16.424418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:16.424423 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fztvp" event={"ID":"98a3da61-90d5-4b7d-842c-683572d6bfbf","Type":"ContainerStarted","Data":"e24332c933091303d03c7234c6403d377854af1e62b6f430585c2de333543d09"} Nov 01 05:09:16.542556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:16.542501 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-fztvp" podStartSLOduration=1.542484926 podStartE2EDuration="1.542484926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:16.541598216 +0000 UTC m=+371.682187134" watchObservedRunningTime="2026-11-01 05:09:16.542484926 +0000 UTC m=+371.683073844" Nov 01 05:09:24.247533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.247499 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:24.250597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.250582 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.306903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.306871 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 05:09:24.316873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.316840 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65zt\" (UniqueName: \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.317017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.316920 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.357519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.357484 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 05:09:24.357689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.357544 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 05:09:24.357740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.357698 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ghcgj\"" Nov 01 05:09:24.358801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.358782 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:24.417761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.417727 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.417934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.417780 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65zt\" (UniqueName: \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.418494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.418475 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.480146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.480111 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65zt\" (UniqueName: \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\") pod \"limitador-limitador-74dfcfb894-9ljw8\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.489955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.489927 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:24.562972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.562939 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:24.937604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:24.937581 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:24.939558 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:09:24.939529 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577ea3e4_20b7_4fce_b66a_d772388bdac6.slice/crio-75faf72943da1348e741eb629bc793c442002671b3d98ecf66a80a03c22f7cce WatchSource:0}: Error finding container 75faf72943da1348e741eb629bc793c442002671b3d98ecf66a80a03c22f7cce: Status 404 returned error can't find the container with id 75faf72943da1348e741eb629bc793c442002671b3d98ecf66a80a03c22f7cce Nov 01 05:09:25.463678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:25.463611 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" event={"ID":"577ea3e4-20b7-4fce-b66a-d772388bdac6","Type":"ContainerStarted","Data":"75faf72943da1348e741eb629bc793c442002671b3d98ecf66a80a03c22f7cce"} Nov 01 05:09:26.951172 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:09:26.951138 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:09:28.441406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.441381 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:09:28.445839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.445823 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:09:28.477819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.477778 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" event={"ID":"577ea3e4-20b7-4fce-b66a-d772388bdac6","Type":"ContainerStarted","Data":"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69"} Nov 01 05:09:28.477950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.477861 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:28.505838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.505807 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gq664\"" Nov 01 05:09:28.507026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.507000 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:09:28.550059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.550022 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47455\" (UniqueName: \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\") pod \"authorino-56c6c9ddd5-6xsvt\" (UID: \"24a81c9e-0f6a-4693-b20e-761962ab9fb0\") " pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:09:28.651328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.651218 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47455\" (UniqueName: \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\") pod \"authorino-56c6c9ddd5-6xsvt\" (UID: \"24a81c9e-0f6a-4693-b20e-761962ab9fb0\") " pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:09:28.678399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.678346 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" podStartSLOduration=1.22270269 podStartE2EDuration="4.678330601s" totalImagesPullingTime="3.455627911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:28.677919914 +0000 UTC m=+383.818508833" watchObservedRunningTime="2026-11-01 05:09:28.678330601 +0000 UTC m=+383.818919520" Nov 01 05:09:28.714655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.714624 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47455\" (UniqueName: \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\") pod \"authorino-56c6c9ddd5-6xsvt\" (UID: \"24a81c9e-0f6a-4693-b20e-761962ab9fb0\") " pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:09:28.761573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:28.761535 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:09:29.130684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:29.130658 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:09:29.132703 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:09:29.132667 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24a81c9e_0f6a_4693_b20e_761962ab9fb0.slice/crio-4b0aa59b8a4d5c724326c444b5dea65c3be73d38c23f57b2a0998c31acd038ad WatchSource:0}: Error finding container 4b0aa59b8a4d5c724326c444b5dea65c3be73d38c23f57b2a0998c31acd038ad: Status 404 returned error can't find the container with id 4b0aa59b8a4d5c724326c444b5dea65c3be73d38c23f57b2a0998c31acd038ad Nov 01 05:09:29.483472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:29.483384 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" event={"ID":"24a81c9e-0f6a-4693-b20e-761962ab9fb0","Type":"ContainerStarted","Data":"4b0aa59b8a4d5c724326c444b5dea65c3be73d38c23f57b2a0998c31acd038ad"} Nov 01 05:09:32.499770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:32.499674 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" event={"ID":"24a81c9e-0f6a-4693-b20e-761962ab9fb0","Type":"ContainerStarted","Data":"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f"} Nov 01 05:09:32.619300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:32.619229 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" podStartSLOduration=1.673885353 podStartE2EDuration="4.619214077s" totalImagesPullingTime="2.945328724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:32.617910643 +0000 UTC m=+387.758499573" watchObservedRunningTime="2026-11-01 05:09:32.619214077 +0000 UTC m=+387.759802997" Nov 01 05:09:39.484408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:39.484372 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:42.536845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:42.536807 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:42.537286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:42.537000 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" podUID="577ea3e4-20b7-4fce-b66a-d772388bdac6" containerName="limitador" containerID="cri-o://1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69" gracePeriod=30 Nov 01 05:09:43.074229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.074201 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:43.174077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.173980 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\" (UniqueName: \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\") pod \"577ea3e4-20b7-4fce-b66a-d772388bdac6\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " Nov 01 05:09:43.174077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.174044 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\" (UniqueName: \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\") pod \"577ea3e4-20b7-4fce-b66a-d772388bdac6\" (UID: \"577ea3e4-20b7-4fce-b66a-d772388bdac6\") " Nov 01 05:09:43.174406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.174386 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file" pod "577ea3e4-20b7-4fce-b66a-d772388bdac6" (UID: "577ea3e4-20b7-4fce-b66a-d772388bdac6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:09:43.176492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.176467 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt" pod "577ea3e4-20b7-4fce-b66a-d772388bdac6" (UID: "577ea3e4-20b7-4fce-b66a-d772388bdac6"). InnerVolumeSpecName "kube-api-access-r65zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:09:43.274583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.274553 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65zt\" (UniqueName: \"kubernetes.io/projected/577ea3e4-20b7-4fce-b66a-d772388bdac6-kube-api-access-r65zt\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:09:43.274583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.274580 4425 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/577ea3e4-20b7-4fce-b66a-d772388bdac6-config-file\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:09:43.549619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.549531 4425 generic.go:356] "Generic (PLEG): container finished" podID="577ea3e4-20b7-4fce-b66a-d772388bdac6" containerID="1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69" exitCode=0 Nov 01 05:09:43.549619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.549574 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" Nov 01 05:09:43.549619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.549611 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" event={"ID":"577ea3e4-20b7-4fce-b66a-d772388bdac6","Type":"ContainerDied","Data":"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69"} Nov 01 05:09:43.550119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.549664 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9ljw8" event={"ID":"577ea3e4-20b7-4fce-b66a-d772388bdac6","Type":"ContainerDied","Data":"75faf72943da1348e741eb629bc793c442002671b3d98ecf66a80a03c22f7cce"} Nov 01 05:09:43.550119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.549688 4425 scope.go:122] "RemoveContainer" containerID="1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69" Nov 01 05:09:43.558010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.557990 4425 scope.go:122] "RemoveContainer" containerID="1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69" Nov 01 05:09:43.558304 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:09:43.558284 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69\": container with ID starting with 1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69 not found: ID does not exist" containerID="1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69" Nov 01 05:09:43.558358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.558315 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69"} err="failed to get container status \"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69\": rpc error: code = NotFound desc = could not find container \"1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69\": container with ID starting with 1c3ded1eb48b1f54508d450d19321062c030831260f859eb705d07a70827ea69 not found: ID does not exist" Nov 01 05:09:43.716373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.716337 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:43.719728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:43.719705 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9ljw8"] Nov 01 05:09:44.971745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:44.971709 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577ea3e4-20b7-4fce-b66a-d772388bdac6" path="/var/lib/kubelet/pods/577ea3e4-20b7-4fce-b66a-d772388bdac6/volumes" Nov 01 05:09:47.731389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.731312 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv"] Nov 01 05:09:47.731746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.731631 4425 cpu_manager.go:414] "containerMap: removing container" podUID="577ea3e4-20b7-4fce-b66a-d772388bdac6" containerName="limitador" Nov 01 05:09:47.731746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.731642 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577ea3e4-20b7-4fce-b66a-d772388bdac6" containerName="limitador" Nov 01 05:09:47.731746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.731696 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="577ea3e4-20b7-4fce-b66a-d772388bdac6" containerName="limitador" Nov 01 05:09:47.734638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.734621 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.788533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.788504 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 05:09:47.788685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.788558 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 05:09:47.812477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.812442 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c1923f2-1a7a-485b-bee5-9f77139043d8-manager-config\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.812628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.812486 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-cert\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.812628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.812511 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6pr\" (UniqueName: \"kubernetes.io/projected/4c1923f2-1a7a-485b-bee5-9f77139043d8-kube-api-access-cn6pr\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.812628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.812606 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-metrics-certs\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.839490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.839454 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bd5vh\"" Nov 01 05:09:47.839490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.839471 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 05:09:47.839675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.839583 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 05:09:47.839675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.839608 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 05:09:47.840825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.840802 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv"] Nov 01 05:09:47.913964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.913935 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c1923f2-1a7a-485b-bee5-9f77139043d8-manager-config\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.914113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.913985 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-cert\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.914113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.914009 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6pr\" (UniqueName: \"kubernetes.io/projected/4c1923f2-1a7a-485b-bee5-9f77139043d8-kube-api-access-cn6pr\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.914199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.914135 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-metrics-certs\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.914839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.914767 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c1923f2-1a7a-485b-bee5-9f77139043d8-manager-config\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.916739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.916714 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-metrics-certs\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.916968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.916947 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c1923f2-1a7a-485b-bee5-9f77139043d8-cert\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:47.982357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:47.982293 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6pr\" (UniqueName: \"kubernetes.io/projected/4c1923f2-1a7a-485b-bee5-9f77139043d8-kube-api-access-cn6pr\") pod \"jobset-controller-manager-956f9bf55-95jwv\" (UID: \"4c1923f2-1a7a-485b-bee5-9f77139043d8\") " pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:48.046065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:48.046028 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:48.437762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:48.437738 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv"] Nov 01 05:09:48.439703 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:09:48.439677 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1923f2_1a7a_485b_bee5_9f77139043d8.slice/crio-a8a74ef6354387ea7a6ff319652b4a250b85987c1c22fefcb1af789b282fd73f WatchSource:0}: Error finding container a8a74ef6354387ea7a6ff319652b4a250b85987c1c22fefcb1af789b282fd73f: Status 404 returned error can't find the container with id a8a74ef6354387ea7a6ff319652b4a250b85987c1c22fefcb1af789b282fd73f Nov 01 05:09:48.574583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:48.574548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" event={"ID":"4c1923f2-1a7a-485b-bee5-9f77139043d8","Type":"ContainerStarted","Data":"a8a74ef6354387ea7a6ff319652b4a250b85987c1c22fefcb1af789b282fd73f"} Nov 01 05:09:53.600018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:53.599984 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" event={"ID":"4c1923f2-1a7a-485b-bee5-9f77139043d8","Type":"ContainerStarted","Data":"75971f33f0c094227e3aaf42bf7ad8bce8df67118d3a6c539dc3654c9a9e1f6e"} Nov 01 05:09:53.600514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:53.600095 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:09:53.718451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:09:53.718400 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" podStartSLOduration=2.602754285 podStartE2EDuration="6.718385509s" totalImagesPullingTime="4.115631224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:53.717748222 +0000 UTC m=+408.858337162" watchObservedRunningTime="2026-11-01 05:09:53.718385509 +0000 UTC m=+408.858974503" Nov 01 05:09:56.946829 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:09:56.946790 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:10:04.610844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:10:04.610815 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-956f9bf55-95jwv" Nov 01 05:10:26.946611 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:10:26.946575 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:10:56.946467 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:10:56.946430 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:11:11.679978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.679944 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:11.682142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.682126 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:11.739769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.739744 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:11.758600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.758572 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmq5x\" (UniqueName: \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\") pod \"authorino-6f6944bcdd-b89n4\" (UID: \"58d50f49-3407-4f42-9ba5-8183114e4a9a\") " pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:11.859936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.859900 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmq5x\" (UniqueName: \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\") pod \"authorino-6f6944bcdd-b89n4\" (UID: \"58d50f49-3407-4f42-9ba5-8183114e4a9a\") " pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:11.922240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.922206 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmq5x\" (UniqueName: \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\") pod \"authorino-6f6944bcdd-b89n4\" (UID: \"58d50f49-3407-4f42-9ba5-8183114e4a9a\") " pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:11.992135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:11.992046 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:12.032071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.032036 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:12.361843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.361820 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:12.364017 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:11:12.363984 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58d50f49_3407_4f42_9ba5_8183114e4a9a.slice/crio-56550fa1baee3b635b338ccaa04c476c50abd5628242f7913c2d8ed23c190d03 WatchSource:0}: Error finding container 56550fa1baee3b635b338ccaa04c476c50abd5628242f7913c2d8ed23c190d03: Status 404 returned error can't find the container with id 56550fa1baee3b635b338ccaa04c476c50abd5628242f7913c2d8ed23c190d03 Nov 01 05:11:12.530497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.530460 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:11:12.533240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.533222 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.565827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.565798 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.565989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.565834 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2v2\" (UniqueName: \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.591088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.591060 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:11:12.640275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.640178 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 05:11:12.666350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.666318 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.666350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.666353 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2v2\" (UniqueName: \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.668886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.668863 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.729082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.729048 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2v2\" (UniqueName: \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\") pod \"authorino-6547cd65c-7v7ck\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.843980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.843949 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:11:12.927714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:12.927681 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" event={"ID":"58d50f49-3407-4f42-9ba5-8183114e4a9a","Type":"ContainerStarted","Data":"56550fa1baee3b635b338ccaa04c476c50abd5628242f7913c2d8ed23c190d03"} Nov 01 05:11:13.214404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:13.214352 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:11:13.216730 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:11:13.216700 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4d8748e_98a9_4e85_a6b5_1ba7d9f02978.slice/crio-4bfdd6c3df5514cc8c142c3da7aa3f1ceedb1d07249fb0d46dffe7e0b2935ec5 WatchSource:0}: Error finding container 4bfdd6c3df5514cc8c142c3da7aa3f1ceedb1d07249fb0d46dffe7e0b2935ec5: Status 404 returned error can't find the container with id 4bfdd6c3df5514cc8c142c3da7aa3f1ceedb1d07249fb0d46dffe7e0b2935ec5 Nov 01 05:11:13.933131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:13.933089 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6547cd65c-7v7ck" event={"ID":"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978","Type":"ContainerStarted","Data":"4bfdd6c3df5514cc8c142c3da7aa3f1ceedb1d07249fb0d46dffe7e0b2935ec5"} Nov 01 05:11:13.934658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:13.934633 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" event={"ID":"58d50f49-3407-4f42-9ba5-8183114e4a9a","Type":"ContainerStarted","Data":"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df"} Nov 01 05:11:13.934750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:13.934699 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" podUID="58d50f49-3407-4f42-9ba5-8183114e4a9a" containerName="authorino" containerID="cri-o://895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df" gracePeriod=30 Nov 01 05:11:14.050915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.050856 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" podStartSLOduration=1.606231106 podStartE2EDuration="3.050839836s" totalImagesPullingTime="1.44460873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:11:14.050356155 +0000 UTC m=+489.190945093" watchObservedRunningTime="2026-11-01 05:11:14.050839836 +0000 UTC m=+489.191428779" Nov 01 05:11:14.286198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.286175 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:14.381816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.381725 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\" (UniqueName: \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\") pod \"58d50f49-3407-4f42-9ba5-8183114e4a9a\" (UID: \"58d50f49-3407-4f42-9ba5-8183114e4a9a\") " Nov 01 05:11:14.383985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.383953 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x" pod "58d50f49-3407-4f42-9ba5-8183114e4a9a" (UID: "58d50f49-3407-4f42-9ba5-8183114e4a9a"). InnerVolumeSpecName "kube-api-access-lmq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:11:14.483023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.482996 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmq5x\" (UniqueName: \"kubernetes.io/projected/58d50f49-3407-4f42-9ba5-8183114e4a9a-kube-api-access-lmq5x\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:11:14.940013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.939972 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6547cd65c-7v7ck" event={"ID":"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978","Type":"ContainerStarted","Data":"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11"} Nov 01 05:11:14.941206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.941172 4425 generic.go:356] "Generic (PLEG): container finished" podID="58d50f49-3407-4f42-9ba5-8183114e4a9a" containerID="895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df" exitCode=0 Nov 01 05:11:14.941343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.941212 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" Nov 01 05:11:14.941343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.941211 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" event={"ID":"58d50f49-3407-4f42-9ba5-8183114e4a9a","Type":"ContainerDied","Data":"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df"} Nov 01 05:11:14.941343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.941292 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b89n4" event={"ID":"58d50f49-3407-4f42-9ba5-8183114e4a9a","Type":"ContainerDied","Data":"56550fa1baee3b635b338ccaa04c476c50abd5628242f7913c2d8ed23c190d03"} Nov 01 05:11:14.941343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.941324 4425 scope.go:122] "RemoveContainer" containerID="895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df" Nov 01 05:11:14.949547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.949531 4425 scope.go:122] "RemoveContainer" containerID="895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df" Nov 01 05:11:14.949828 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:11:14.949808 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df\": container with ID starting with 895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df not found: ID does not exist" containerID="895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df" Nov 01 05:11:14.949892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:14.949837 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df"} err="failed to get container status \"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df\": rpc error: code = NotFound desc = could not find container \"895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df\": container with ID starting with 895ae9a633fc614470aa01167475079fd83135659a4fe950247a1e72a31318df not found: ID does not exist" Nov 01 05:11:15.060945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.060881 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6547cd65c-7v7ck" podStartSLOduration=1.873650475 podStartE2EDuration="3.06086215s" totalImagesPullingTime="1.187211675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:11:15.058815956 +0000 UTC m=+490.199404877" watchObservedRunningTime="2026-11-01 05:11:15.06086215 +0000 UTC m=+490.201451070" Nov 01 05:11:15.088161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.088125 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:11:15.088382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.088361 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" podUID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" containerName="authorino" containerID="cri-o://9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f" gracePeriod=30 Nov 01 05:11:15.181927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.181851 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:15.183436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.183415 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b89n4"] Nov 01 05:11:15.351223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.351196 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:11:15.390377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.390345 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\" (UniqueName: \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\") pod \"24a81c9e-0f6a-4693-b20e-761962ab9fb0\" (UID: \"24a81c9e-0f6a-4693-b20e-761962ab9fb0\") " Nov 01 05:11:15.392564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.392537 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455" pod "24a81c9e-0f6a-4693-b20e-761962ab9fb0" (UID: "24a81c9e-0f6a-4693-b20e-761962ab9fb0"). InnerVolumeSpecName "kube-api-access-47455". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:11:15.490982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.490908 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47455\" (UniqueName: \"kubernetes.io/projected/24a81c9e-0f6a-4693-b20e-761962ab9fb0-kube-api-access-47455\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:11:15.946731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.946687 4425 generic.go:356] "Generic (PLEG): container finished" podID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" containerID="9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f" exitCode=0 Nov 01 05:11:15.946731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.946729 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" Nov 01 05:11:15.947229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.946779 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" event={"ID":"24a81c9e-0f6a-4693-b20e-761962ab9fb0","Type":"ContainerDied","Data":"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f"} Nov 01 05:11:15.947229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.946806 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6xsvt" event={"ID":"24a81c9e-0f6a-4693-b20e-761962ab9fb0","Type":"ContainerDied","Data":"4b0aa59b8a4d5c724326c444b5dea65c3be73d38c23f57b2a0998c31acd038ad"} Nov 01 05:11:15.947229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.946822 4425 scope.go:122] "RemoveContainer" containerID="9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f" Nov 01 05:11:15.955576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.955550 4425 scope.go:122] "RemoveContainer" containerID="9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f" Nov 01 05:11:15.955849 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:11:15.955829 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f\": container with ID starting with 9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f not found: ID does not exist" containerID="9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f" Nov 01 05:11:15.955957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:15.955855 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f"} err="failed to get container status \"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f\": rpc error: code = NotFound desc = could not find container \"9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f\": container with ID starting with 9a36202b0eabf3d59abda2b7a5801a0ccf0547f5815d83163c5df797f299931f not found: ID does not exist" Nov 01 05:11:16.112962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:16.112933 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:11:16.163774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:16.163740 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6xsvt"] Nov 01 05:11:16.970571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:16.970501 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" path="/var/lib/kubelet/pods/24a81c9e-0f6a-4693-b20e-761962ab9fb0/volumes" Nov 01 05:11:16.970952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:11:16.970932 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d50f49-3407-4f42-9ba5-8183114e4a9a" path="/var/lib/kubelet/pods/58d50f49-3407-4f42-9ba5-8183114e4a9a/volumes" Nov 01 05:11:26.946417 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:11:26.946379 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:11:56.947473 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:11:56.947436 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:12:26.946311 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:12:26.946247 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:12:30.413468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413437 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413770 4425 cpu_manager.go:414] "containerMap: removing container" podUID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" containerName="authorino" Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413781 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" containerName="authorino" Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413793 4425 cpu_manager.go:414] "containerMap: removing container" podUID="58d50f49-3407-4f42-9ba5-8183114e4a9a" containerName="authorino" Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413798 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d50f49-3407-4f42-9ba5-8183114e4a9a" containerName="authorino" Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413849 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a81c9e-0f6a-4693-b20e-761962ab9fb0" containerName="authorino" Nov 01 05:12:30.415771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.413858 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d50f49-3407-4f42-9ba5-8183114e4a9a" containerName="authorino" Nov 01 05:12:30.416777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.416762 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.473042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.473013 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 05:12:30.474778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.474753 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:12:30.523247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.523219 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 05:12:30.523428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.523359 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 05:12:30.523491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.523473 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-zzp6h\"" Nov 01 05:12:30.523557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.523542 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 05:12:30.531957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.531938 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvnw\" (UniqueName: \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.532052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.531978 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.632864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.632826 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvnw\" (UniqueName: \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.633024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.632871 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.635475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.635451 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.694886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.694810 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvnw\" (UniqueName: \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\") pod \"ai-gateway-controller-7d85675f98-ffpn8\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:30.729729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:30.729701 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:31.099563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:31.099526 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:12:31.102141 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:12:31.102116 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6032c1d1_116e_4fe6_8c51_3e3a56d768be.slice/crio-b2e4b433295471b35e7975e3bff00f63d7e8f3c2a6b02985b11429509c4c7fbb WatchSource:0}: Error finding container b2e4b433295471b35e7975e3bff00f63d7e8f3c2a6b02985b11429509c4c7fbb: Status 404 returned error can't find the container with id b2e4b433295471b35e7975e3bff00f63d7e8f3c2a6b02985b11429509c4c7fbb Nov 01 05:12:31.258569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:31.258538 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" event={"ID":"6032c1d1-116e-4fe6-8c51-3e3a56d768be","Type":"ContainerStarted","Data":"b2e4b433295471b35e7975e3bff00f63d7e8f3c2a6b02985b11429509c4c7fbb"} Nov 01 05:12:34.274820 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:34.274727 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" event={"ID":"6032c1d1-116e-4fe6-8c51-3e3a56d768be","Type":"ContainerStarted","Data":"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4"} Nov 01 05:12:34.275211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:34.274837 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:34.392788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:34.392735 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" podStartSLOduration=1.537222375 podStartE2EDuration="4.39272242s" totalImagesPullingTime="2.855500045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:12:34.392352844 +0000 UTC m=+569.532941771" watchObservedRunningTime="2026-11-01 05:12:34.39272242 +0000 UTC m=+569.533311338" Nov 01 05:12:38.389331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.389294 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 05:12:38.392894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.392870 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.406536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.406496 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.406685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.406574 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2m9\" (UniqueName: \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.451065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.451030 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 05:12:38.496594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.496561 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 05:12:38.496754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.496630 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gj8f2\"" Nov 01 05:12:38.507119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.507091 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.507228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.507145 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2m9\" (UniqueName: \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.508105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.508079 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.549160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.549124 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 05:12:38.552830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.552810 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.569150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.569127 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2m9\" (UniqueName: \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\") pod \"payload-pre-processing-7b98bb678c-j678p\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.599779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.599746 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 05:12:38.608320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.608294 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.608474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.608349 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5kk\" (UniqueName: \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.709734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.709641 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5kk\" (UniqueName: \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.709734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.709733 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.710399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.710382 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.711584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.711565 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:38.772202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.772168 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5kk\" (UniqueName: \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\") pod \"payload-processing-55fbbbf9dc-rfb94\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:38.864925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:38.864886 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:39.085786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:39.085761 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 05:12:39.088580 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:12:39.088554 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32e5869_ff99_49e7_ad8e_e032c4d44356.slice/crio-f1941e749c3ae85d22abd17687c2dd5ce89fd344a16f11323ecfbaeb2ce33380 WatchSource:0}: Error finding container f1941e749c3ae85d22abd17687c2dd5ce89fd344a16f11323ecfbaeb2ce33380: Status 404 returned error can't find the container with id f1941e749c3ae85d22abd17687c2dd5ce89fd344a16f11323ecfbaeb2ce33380 Nov 01 05:12:39.243245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:39.243218 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 05:12:39.245722 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:12:39.245692 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdd9883_e2d7_48e4_b283_f6f3137e7065.slice/crio-bc0e5c891c4619296e582cf8eb7ad33e9e40f67eaf2dbf0e8ac3633336373f15 WatchSource:0}: Error finding container bc0e5c891c4619296e582cf8eb7ad33e9e40f67eaf2dbf0e8ac3633336373f15: Status 404 returned error can't find the container with id bc0e5c891c4619296e582cf8eb7ad33e9e40f67eaf2dbf0e8ac3633336373f15 Nov 01 05:12:39.297374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:39.297340 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" event={"ID":"ecdd9883-e2d7-48e4-b283-f6f3137e7065","Type":"ContainerStarted","Data":"bc0e5c891c4619296e582cf8eb7ad33e9e40f67eaf2dbf0e8ac3633336373f15"} Nov 01 05:12:39.298561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:39.298535 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" event={"ID":"b32e5869-ff99-49e7-ad8e-e032c4d44356","Type":"ContainerStarted","Data":"f1941e749c3ae85d22abd17687c2dd5ce89fd344a16f11323ecfbaeb2ce33380"} Nov 01 05:12:42.314377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.314347 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" event={"ID":"ecdd9883-e2d7-48e4-b283-f6f3137e7065","Type":"ContainerStarted","Data":"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45"} Nov 01 05:12:42.314847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.314434 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:42.315851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.315831 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" event={"ID":"b32e5869-ff99-49e7-ad8e-e032c4d44356","Type":"ContainerStarted","Data":"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10"} Nov 01 05:12:42.315981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.315952 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:42.433135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.433073 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" podStartSLOduration=1.719879494 podStartE2EDuration="4.43305669s" totalImagesPullingTime="2.713177196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:12:42.431971037 +0000 UTC m=+577.572559981" watchObservedRunningTime="2026-11-01 05:12:42.43305669 +0000 UTC m=+577.573645651" Nov 01 05:12:42.549740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:42.549687 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" podStartSLOduration=1.780309898 podStartE2EDuration="4.549672861s" totalImagesPullingTime="2.769362963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:12:42.548630015 +0000 UTC m=+577.689218947" watchObservedRunningTime="2026-11-01 05:12:42.549672861 +0000 UTC m=+577.690261780" Nov 01 05:12:45.282273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.282223 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:45.638741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.638707 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-559cb95fbf-czdwv"] Nov 01 05:12:45.642347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.642329 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.672427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.672387 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/9e4d7b1c-a9bf-439c-b355-4149893982d5-kube-api-access-f5llv\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.672616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.672508 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d7b1c-a9bf-439c-b355-4149893982d5-metrics-certs\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.699884 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.699848 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-559cb95fbf-czdwv"] Nov 01 05:12:45.773823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.773783 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d7b1c-a9bf-439c-b355-4149893982d5-metrics-certs\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.774030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.773870 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/9e4d7b1c-a9bf-439c-b355-4149893982d5-kube-api-access-f5llv\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.776552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.776527 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d7b1c-a9bf-439c-b355-4149893982d5-metrics-certs\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.836514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.836478 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5llv\" (UniqueName: \"kubernetes.io/projected/9e4d7b1c-a9bf-439c-b355-4149893982d5-kube-api-access-f5llv\") pod \"ai-gateway-controller-559cb95fbf-czdwv\" (UID: \"9e4d7b1c-a9bf-439c-b355-4149893982d5\") " pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:45.954989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:45.954901 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:46.328041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:46.327843 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-559cb95fbf-czdwv"] Nov 01 05:12:46.330529 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:12:46.330505 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4d7b1c_a9bf_439c_b355_4149893982d5.slice/crio-fb819e43d376b62fa91a70b4dfac4a06742a00769165b6b49ecb4819f90b211c WatchSource:0}: Error finding container fb819e43d376b62fa91a70b4dfac4a06742a00769165b6b49ecb4819f90b211c: Status 404 returned error can't find the container with id fb819e43d376b62fa91a70b4dfac4a06742a00769165b6b49ecb4819f90b211c Nov 01 05:12:47.338432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:47.338395 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" event={"ID":"9e4d7b1c-a9bf-439c-b355-4149893982d5","Type":"ContainerStarted","Data":"282782d071b6951ff1a4a3c38386fc26e3d58119beaa7f5187d5240237319348"} Nov 01 05:12:47.338817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:47.338448 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" event={"ID":"9e4d7b1c-a9bf-439c-b355-4149893982d5","Type":"ContainerStarted","Data":"fb819e43d376b62fa91a70b4dfac4a06742a00769165b6b49ecb4819f90b211c"} Nov 01 05:12:47.338817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:47.338520 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:47.457340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:47.457216 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" podStartSLOduration=1.669267757 podStartE2EDuration="2.457200615s" totalImagesPullingTime="787.932858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:12:47.456214451 +0000 UTC m=+582.596803389" watchObservedRunningTime="2026-11-01 05:12:47.457200615 +0000 UTC m=+582.597789533" Nov 01 05:12:53.323062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:53.323033 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 05:12:53.323494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:53.323105 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 05:12:56.947474 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:12:56.947436 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:12:58.345359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.345331 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 05:12:58.516348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.516314 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:12:58.516547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.516523 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" podUID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" containerName="manager" containerID="cri-o://828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4" gracePeriod=30 Nov 01 05:12:58.764963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.764940 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:58.896622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.896537 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\" (UniqueName: \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\") pod \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " Nov 01 05:12:58.896622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.896610 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\") pod \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\" (UID: \"6032c1d1-116e-4fe6-8c51-3e3a56d768be\") " Nov 01 05:12:58.898991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.898957 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs" pod "6032c1d1-116e-4fe6-8c51-3e3a56d768be" (UID: "6032c1d1-116e-4fe6-8c51-3e3a56d768be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:12:58.899137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.898999 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw" pod "6032c1d1-116e-4fe6-8c51-3e3a56d768be" (UID: "6032c1d1-116e-4fe6-8c51-3e3a56d768be"). InnerVolumeSpecName "kube-api-access-jlvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:12:58.997650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.997619 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvnw\" (UniqueName: \"kubernetes.io/projected/6032c1d1-116e-4fe6-8c51-3e3a56d768be-kube-api-access-jlvnw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:12:58.997650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:58.997648 4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6032c1d1-116e-4fe6-8c51-3e3a56d768be-metrics-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:12:59.390335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.390306 4425 generic.go:356] "Generic (PLEG): container finished" podID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" containerID="828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4" exitCode=0 Nov 01 05:12:59.390741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.390354 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" Nov 01 05:12:59.390741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.390401 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" event={"ID":"6032c1d1-116e-4fe6-8c51-3e3a56d768be","Type":"ContainerDied","Data":"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4"} Nov 01 05:12:59.390741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.390452 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-ffpn8" event={"ID":"6032c1d1-116e-4fe6-8c51-3e3a56d768be","Type":"ContainerDied","Data":"b2e4b433295471b35e7975e3bff00f63d7e8f3c2a6b02985b11429509c4c7fbb"} Nov 01 05:12:59.390741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.390473 4425 scope.go:122] "RemoveContainer" containerID="828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4" Nov 01 05:12:59.398669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.398649 4425 scope.go:122] "RemoveContainer" containerID="828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4" Nov 01 05:12:59.399094 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:12:59.399059 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4\": container with ID starting with 828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4 not found: ID does not exist" containerID="828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4" Nov 01 05:12:59.399210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.399101 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4"} err="failed to get container status \"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4\": rpc error: code = NotFound desc = could not find container \"828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4\": container with ID starting with 828a1633e3fdc0f122bfd99bc937bbd0bcf13154e9bac7ae55fa46ce95c681f4 not found: ID does not exist" Nov 01 05:12:59.557203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.557166 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:12:59.608029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:12:59.607999 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-ffpn8"] Nov 01 05:13:00.971961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:00.971927 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" path="/var/lib/kubelet/pods/6032c1d1-116e-4fe6-8c51-3e3a56d768be/volumes" Nov 01 05:13:20.539841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.539805 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj"] Nov 01 05:13:20.540335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.540166 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" containerName="manager" Nov 01 05:13:20.540335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.540176 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" containerName="manager" Nov 01 05:13:20.540335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.540277 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6032c1d1-116e-4fe6-8c51-3e3a56d768be" containerName="manager" Nov 01 05:13:20.543348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.543330 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.647736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.647706 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 05:13:20.647736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.647735 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w97n\"" Nov 01 05:13:20.648002 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.647735 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 05:13:20.648002 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.647924 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 05:13:20.649335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.649315 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj"] Nov 01 05:13:20.692436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692390 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.692436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692437 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rc6v\" (UniqueName: \"kubernetes.io/projected/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-kube-api-access-8rc6v\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.692620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692521 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.692620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692592 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.692716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692654 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.692716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.692703 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794136 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794184 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794227 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794306 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794386 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794419 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rc6v\" (UniqueName: \"kubernetes.io/projected/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-kube-api-access-8rc6v\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794661 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794780 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.794860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.794818 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.796834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.796813 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.797106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.797082 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:20.856662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:20.856627 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rc6v\" (UniqueName: \"kubernetes.io/projected/fc8a7b4f-056c-4649-9b91-f4f625e9e24c-kube-api-access-8rc6v\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mf4nj\" (UID: \"fc8a7b4f-056c-4649-9b91-f4f625e9e24c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:21.155799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:21.155765 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:21.532636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:21.532611 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj"] Nov 01 05:13:21.534687 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:21.534663 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8a7b4f_056c_4649_9b91_f4f625e9e24c.slice/crio-7ca22d84610fb54a478d671b7572b5cea28114503268e7ca3ca8126e72660903 WatchSource:0}: Error finding container 7ca22d84610fb54a478d671b7572b5cea28114503268e7ca3ca8126e72660903: Status 404 returned error can't find the container with id 7ca22d84610fb54a478d671b7572b5cea28114503268e7ca3ca8126e72660903 Nov 01 05:13:22.492531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:22.492490 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" event={"ID":"fc8a7b4f-056c-4649-9b91-f4f625e9e24c","Type":"ContainerStarted","Data":"7ca22d84610fb54a478d671b7572b5cea28114503268e7ca3ca8126e72660903"} Nov 01 05:13:23.500460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:23.500363 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" event={"ID":"fc8a7b4f-056c-4649-9b91-f4f625e9e24c","Type":"ContainerStarted","Data":"6615b95784f01feb63eed7869ea6588fd4e09dc888006113ad9e00e6e2c51c18"} Nov 01 05:13:23.621061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:23.621004 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" podStartSLOduration=1.92023313 podStartE2EDuration="3.620990904s" totalImagesPullingTime="1.700757774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:23.620319666 +0000 UTC m=+618.760908584" watchObservedRunningTime="2026-11-01 05:13:23.620990904 +0000 UTC m=+618.761579817" Nov 01 05:13:24.156290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:24.156233 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:25.510630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.510598 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:13:25.514066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.514050 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.537953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.537921 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.537953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.537958 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.537983 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538065 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538301 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqwp\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538350 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538395 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538415 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.538489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.538477 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.561495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.561458 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:13:25.564810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.564793 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.567440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.567401 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 05:13:25.613115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.613081 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:13:25.620004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.619974 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:13:25.639374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639335 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639378 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639406 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.639625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639448 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639565 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.639625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639591 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639718 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639759 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639766 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639793 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639815 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.639837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639820 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639868 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkl5c\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.639916 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqwp\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640034 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640040 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640081 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.640109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640094 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.640471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640202 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640233 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640269 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.640632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.640481 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.642382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.642357 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.642522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.642507 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.659146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.659114 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\"" Nov 01 05:13:25.702684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.702654 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.702838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.702714 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqwp\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\") pod \"maas-default-gateway-openshift-default-58d898bd55-tn2cx\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.740968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.740932 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.740990 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741006 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkl5c\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741048 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741067 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741084 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741104 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741157 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741231 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741490 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741704 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741708 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741756 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.741851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.741796 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.743894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.743863 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.744064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.744047 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.803489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.803406 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.803609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.803534 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkl5c\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\") pod \"maas-default-gateway-openshift-default-58d898bd55-7g89g\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:25.826841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.826818 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:25.881215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:25.880801 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:26.252419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:26.252397 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:13:26.253882 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:26.253851 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a256d4_691c_4148_bf6a_f9cf0d04e8fe.slice/crio-73c830b3fff78a4fe2b33193de53a4823b593a1361b4db3f2854c55dd469761a WatchSource:0}: Error finding container 73c830b3fff78a4fe2b33193de53a4823b593a1361b4db3f2854c55dd469761a: Status 404 returned error can't find the container with id 73c830b3fff78a4fe2b33193de53a4823b593a1361b4db3f2854c55dd469761a Nov 01 05:13:26.311821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:26.311794 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:13:26.313531 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:26.313508 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26dc12f_9951_4c09_8fa7_52e84ab40c20.slice/crio-ad3bc9144b364001262bfc7067400d4a6baa4d67bf96c4703d468a05d5bf366a WatchSource:0}: Error finding container ad3bc9144b364001262bfc7067400d4a6baa4d67bf96c4703d468a05d5bf366a: Status 404 returned error can't find the container with id ad3bc9144b364001262bfc7067400d4a6baa4d67bf96c4703d468a05d5bf366a Nov 01 05:13:26.515246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:26.515152 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" event={"ID":"90a256d4-691c-4148-bf6a-f9cf0d04e8fe","Type":"ContainerStarted","Data":"73c830b3fff78a4fe2b33193de53a4823b593a1361b4db3f2854c55dd469761a"} Nov 01 05:13:26.516488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:26.516462 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" event={"ID":"d26dc12f-9951-4c09-8fa7-52e84ab40c20","Type":"ContainerStarted","Data":"ad3bc9144b364001262bfc7067400d4a6baa4d67bf96c4703d468a05d5bf366a"} Nov 01 05:13:26.946936 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:13:26.946902 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:13:30.617558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.617053 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:30.617558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.617159 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:30.617558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.617215 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:30.952220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.952188 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:30.952342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.952314 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:30.952386 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:30.952368 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:31.158278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.157712 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:31.177306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.175832 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:31.282372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.282280 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5"] Nov 01 05:13:31.286247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.286221 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.332045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.332013 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5"] Nov 01 05:13:31.355278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.351534 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 05:13:31.406668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406625 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a5d029-b30e-40a5-ba11-40e5d13560bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.406870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406697 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.406870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406794 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.406870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406856 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.407054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406907 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.407054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.406974 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qr2\" (UniqueName: \"kubernetes.io/projected/18a5d029-b30e-40a5-ba11-40e5d13560bc-kube-api-access-w8qr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508010 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qr2\" (UniqueName: \"kubernetes.io/projected/18a5d029-b30e-40a5-ba11-40e5d13560bc-kube-api-access-w8qr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508120 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a5d029-b30e-40a5-ba11-40e5d13560bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508170 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508199 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508235 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508307 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508787 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.508946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.508918 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.509066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.509046 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.510866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.510843 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a5d029-b30e-40a5-ba11-40e5d13560bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.511005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.510991 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a5d029-b30e-40a5-ba11-40e5d13560bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.577240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.576324 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" event={"ID":"90a256d4-691c-4148-bf6a-f9cf0d04e8fe","Type":"ContainerStarted","Data":"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f"} Nov 01 05:13:31.597275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.596998 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" event={"ID":"d26dc12f-9951-4c09-8fa7-52e84ab40c20","Type":"ContainerStarted","Data":"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c"} Nov 01 05:13:31.598996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.598474 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qr2\" (UniqueName: \"kubernetes.io/projected/18a5d029-b30e-40a5-ba11-40e5d13560bc-kube-api-access-w8qr2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5\" (UID: \"18a5d029-b30e-40a5-ba11-40e5d13560bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.609219 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.609009 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:31.611109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.611090 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mf4nj" Nov 01 05:13:31.822688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.822632 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podStartSLOduration=2.127031061 podStartE2EDuration="6.82261643s" totalImagesPullingTime="4.695585369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:31.700469321 +0000 UTC m=+626.841058260" watchObservedRunningTime="2026-11-01 05:13:31.82261643 +0000 UTC m=+626.963205347" Nov 01 05:13:31.827947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.827897 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:31.891302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.883625 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:31.898408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.891622 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:31.898408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:31.891686 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:32.038279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.037214 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podStartSLOduration=2.736892292 podStartE2EDuration="7.037194544s" totalImagesPullingTime="4.300302252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:31.951359142 +0000 UTC m=+627.091948084" watchObservedRunningTime="2026-11-01 05:13:32.037194544 +0000 UTC m=+627.177783466" Nov 01 05:13:32.061280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.046120 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5"] Nov 01 05:13:32.069277 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:32.066223 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a5d029_b30e_40a5_ba11_40e5d13560bc.slice/crio-44a6f9ce70edc87788fa7fedf6d2c9375e29fba3309154a4251b6fc6070c7bf5 WatchSource:0}: Error finding container 44a6f9ce70edc87788fa7fedf6d2c9375e29fba3309154a4251b6fc6070c7bf5: Status 404 returned error can't find the container with id 44a6f9ce70edc87788fa7fedf6d2c9375e29fba3309154a4251b6fc6070c7bf5 Nov 01 05:13:32.602221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.602186 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" event={"ID":"18a5d029-b30e-40a5-ba11-40e5d13560bc","Type":"ContainerStarted","Data":"44a6f9ce70edc87788fa7fedf6d2c9375e29fba3309154a4251b6fc6070c7bf5"} Nov 01 05:13:32.829458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.829423 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:32.829840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.829475 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:32.882329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.881666 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:32.882329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:32.881727 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:33.609296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.609187 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" event={"ID":"18a5d029-b30e-40a5-ba11-40e5d13560bc","Type":"ContainerStarted","Data":"374f4e9f9624c200991b3818bbfcc00c2cd1114ed64ef0eb099711a47a8b5e17"} Nov 01 05:13:33.609632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.609603 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:33.735359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.735295 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" podStartSLOduration=2.011931624 podStartE2EDuration="2.735279568s" totalImagesPullingTime="723.347944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:33.73364955 +0000 UTC m=+628.874238510" watchObservedRunningTime="2026-11-01 05:13:33.735279568 +0000 UTC m=+628.875868486" Nov 01 05:13:33.827513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.827476 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:33.827689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.827537 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:33.884339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.882702 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:33.884339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:33.882764 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:34.828159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:34.828117 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:34.828366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:34.828173 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:34.885279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:34.885211 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:34.885787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:34.885310 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:35.827342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:35.827303 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:35.827530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:35.827371 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:35.881839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:35.881705 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:35.881839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:35.881775 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:36.828237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:36.828200 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:36.828744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:36.828296 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:36.881383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:36.881351 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 01 05:13:36.881559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:36.881410 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 01 05:13:37.827595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:37.827543 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 01 05:13:37.827818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:37.827637 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 01 05:13:38.196927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:38.196407 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:38.197436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:38.197160 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:38.198320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:38.198296 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:13:39.868500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:39.868457 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tn2cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:13:39.869138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:39.869091 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:13:39.929434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:39.929402 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr"] Nov 01 05:13:39.933137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:39.933107 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:39.987315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:39.987281 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 05:13:40.035831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.035795 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr"] Nov 01 05:13:40.048764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048735 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.048951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048772 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46km\" (UniqueName: \"kubernetes.io/projected/61c1575d-d0b3-443d-8b98-812b59888456-kube-api-access-p46km\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.048951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048796 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.048951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048883 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c1575d-d0b3-443d-8b98-812b59888456-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.048951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048901 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.048951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.048921 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149595 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c1575d-d0b3-443d-8b98-812b59888456-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149651 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149686 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149741 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149863 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46km\" (UniqueName: \"kubernetes.io/projected/61c1575d-d0b3-443d-8b98-812b59888456-kube-api-access-p46km\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.149985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.149911 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.150136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.150079 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.150616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.150588 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.150759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.150704 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.152913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.152884 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c1575d-d0b3-443d-8b98-812b59888456-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.153087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.153066 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c1575d-d0b3-443d-8b98-812b59888456-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.216594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.216561 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46km\" (UniqueName: \"kubernetes.io/projected/61c1575d-d0b3-443d-8b98-812b59888456-kube-api-access-p46km\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr\" (UID: \"61c1575d-d0b3-443d-8b98-812b59888456\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.246494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.246461 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:40.433148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.433049 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:40.433367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.433347 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:40.433985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.433969 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:13:40.636337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.636308 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:13:40.640827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.640805 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.670825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.670802 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr"] Nov 01 05:13:40.672020 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:40.672000 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c1575d_d0b3_443d_8b98_812b59888456.slice/crio-c531067cd97a838c850e396e6c8ec3e9341c1b6694c290c378730aeb08accb2e WatchSource:0}: Error finding container c531067cd97a838c850e396e6c8ec3e9341c1b6694c290c378730aeb08accb2e: Status 404 returned error can't find the container with id c531067cd97a838c850e396e6c8ec3e9341c1b6694c290c378730aeb08accb2e Nov 01 05:13:40.673398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.673380 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:13:40.769847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.769764 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770059 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770277 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770411 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v98t6\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770553 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770667 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.770783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.770770 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.771032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.771019 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.771148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.771133 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.778637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.778616 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:13:40.872672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872634 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.872672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872678 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872705 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872727 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872794 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872821 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v98t6\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872861 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872879 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.872894 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.873166 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.873233 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.873410 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.873480 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.873678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.873607 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.875508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.875481 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.875746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.875726 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.935118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.935083 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.935273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.935155 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v98t6\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\") pod \"maas-default-gateway-openshift-default-58d898bd55-94m8z\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:40.955084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:40.955048 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:41.380871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.380847 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:13:41.382713 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:41.382688 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba2e5a0_c74e_45c6_a8aa_6dc5a3ca04c0.slice/crio-50d93cafcf30bdb3b7424cffa27ddb1bdd168bd70d31ab4d6bfea6d3c51d7742 WatchSource:0}: Error finding container 50d93cafcf30bdb3b7424cffa27ddb1bdd168bd70d31ab4d6bfea6d3c51d7742: Status 404 returned error can't find the container with id 50d93cafcf30bdb3b7424cffa27ddb1bdd168bd70d31ab4d6bfea6d3c51d7742 Nov 01 05:13:41.387203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.387172 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:41.387299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.387282 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:41.387346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.387323 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:13:41.610506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.610472 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:41.623814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.623789 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:41.652059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.651961 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" event={"ID":"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0","Type":"ContainerStarted","Data":"5ee2d7e73c34c25c8fbaf3690252006fe78daebf5e2570114e5df23c806f4146"} Nov 01 05:13:41.652059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.652030 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" event={"ID":"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0","Type":"ContainerStarted","Data":"50d93cafcf30bdb3b7424cffa27ddb1bdd168bd70d31ab4d6bfea6d3c51d7742"} Nov 01 05:13:41.653890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.653863 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" event={"ID":"61c1575d-d0b3-443d-8b98-812b59888456","Type":"ContainerStarted","Data":"3afb61893f320815f840e7a26c681cb4e39497946381b72d88ef066fa14f8ee1"} Nov 01 05:13:41.654012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.653910 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" event={"ID":"61c1575d-d0b3-443d-8b98-812b59888456","Type":"ContainerStarted","Data":"c531067cd97a838c850e396e6c8ec3e9341c1b6694c290c378730aeb08accb2e"} Nov 01 05:13:41.670714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.670684 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5" Nov 01 05:13:41.966340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.955358 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:41.992080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:41.992033 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podStartSLOduration=1.992020638 podStartE2EDuration="1.992020638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:41.991647876 +0000 UTC m=+637.132236815" watchObservedRunningTime="2026-11-01 05:13:41.992020638 +0000 UTC m=+637.132609548" Nov 01 05:13:42.112752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:42.112701 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" podStartSLOduration=2.395728687 podStartE2EDuration="3.112687553s" totalImagesPullingTime="716.958866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:42.111760142 +0000 UTC m=+637.252349061" watchObservedRunningTime="2026-11-01 05:13:42.112687553 +0000 UTC m=+637.253276487" Nov 01 05:13:42.260478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:42.260374 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:42.964277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:42.961721 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:42.964277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:42.961798 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:43.964280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:43.961367 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:43.964280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:43.961442 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:44.959280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:44.956636 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:44.959280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:44.956698 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:45.559276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.550452 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz"] Nov 01 05:13:45.577053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.575214 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.615114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.614562 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz"] Nov 01 05:13:45.630466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.630436 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 05:13:45.722373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.722328 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.722655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.722636 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.722833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.722817 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.722956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.722940 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cj5\" (UniqueName: \"kubernetes.io/projected/27bad948-9ce6-42ff-b5e2-3deaa8d02069-kube-api-access-b8cj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.723091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.723077 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.723285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.723234 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824652 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824704 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824763 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824793 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cj5\" (UniqueName: \"kubernetes.io/projected/27bad948-9ce6-42ff-b5e2-3deaa8d02069-kube-api-access-b8cj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824844 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.824903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.824881 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.825442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.825412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.825876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.825844 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.826022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.826000 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.828172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.828153 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bad948-9ce6-42ff-b5e2-3deaa8d02069-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.829685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.829663 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bad948-9ce6-42ff-b5e2-3deaa8d02069-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.888399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.888374 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cj5\" (UniqueName: \"kubernetes.io/projected/27bad948-9ce6-42ff-b5e2-3deaa8d02069-kube-api-access-b8cj5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz\" (UID: \"27bad948-9ce6-42ff-b5e2-3deaa8d02069\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.888730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.888715 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:45.964280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.955719 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:45.964280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:45.955782 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:46.317412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:46.317364 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz"] Nov 01 05:13:46.318661 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:13:46.318618 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27bad948_9ce6_42ff_b5e2_3deaa8d02069.slice/crio-4b56edb6d1878694062e024f7b2c00d4b78d5d9ef82fbf737048eb96d6c98646 WatchSource:0}: Error finding container 4b56edb6d1878694062e024f7b2c00d4b78d5d9ef82fbf737048eb96d6c98646: Status 404 returned error can't find the container with id 4b56edb6d1878694062e024f7b2c00d4b78d5d9ef82fbf737048eb96d6c98646 Nov 01 05:13:46.734619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:46.734512 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" event={"ID":"27bad948-9ce6-42ff-b5e2-3deaa8d02069","Type":"ContainerStarted","Data":"4b56edb6d1878694062e024f7b2c00d4b78d5d9ef82fbf737048eb96d6c98646"} Nov 01 05:13:46.959282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:46.958698 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:46.959282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:46.958757 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:47.742289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:47.741833 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" event={"ID":"27bad948-9ce6-42ff-b5e2-3deaa8d02069","Type":"ContainerStarted","Data":"36a43cc13bb9b3e87b9d501743dc01e83573f21b8e5aa8e11744683867a90238"} Nov 01 05:13:47.890279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:47.889290 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:47.960279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:47.956443 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 05:13:47.960279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:47.956506 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 05:13:49.956695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:49.956640 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:13:49.957128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:49.956716 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:13:50.247124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.247021 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:50.259777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.259746 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:50.380885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.380807 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" podStartSLOduration=4.569909189 podStartE2EDuration="5.380787551s" totalImagesPullingTime="810.878362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:47.870127855 +0000 UTC m=+643.010716775" watchObservedRunningTime="2026-11-01 05:13:50.380787551 +0000 UTC m=+645.521376474" Nov 01 05:13:50.779675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.779640 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr" Nov 01 05:13:50.998465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.998416 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:13:50.998827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:50.998483 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:13:51.549644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:51.549612 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:51.550725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:51.550693 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:51.551892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:51.551815 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:13:55.889414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:55.889380 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:55.902199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:55.902177 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:56.842805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:13:56.842778 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" Nov 01 05:13:56.946900 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:13:56.946861 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:14:10.801022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:10.800988 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-847cf6dc9d-lk8pc"] Nov 01 05:14:10.810808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:10.810777 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:10.865503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:10.865477 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-847cf6dc9d-lk8pc"] Nov 01 05:14:10.910354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:10.910320 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/085b04bc-51a6-4928-815d-f38fe27aea3d-tls-cert\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:10.910513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:10.910433 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfblb\" (UniqueName: \"kubernetes.io/projected/085b04bc-51a6-4928-815d-f38fe27aea3d-kube-api-access-hfblb\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.011595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.011556 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/085b04bc-51a6-4928-815d-f38fe27aea3d-tls-cert\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.011799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.011623 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfblb\" (UniqueName: \"kubernetes.io/projected/085b04bc-51a6-4928-815d-f38fe27aea3d-kube-api-access-hfblb\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.014241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.014202 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/085b04bc-51a6-4928-815d-f38fe27aea3d-tls-cert\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.079370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.079341 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfblb\" (UniqueName: \"kubernetes.io/projected/085b04bc-51a6-4928-815d-f38fe27aea3d-kube-api-access-hfblb\") pod \"authorino-847cf6dc9d-lk8pc\" (UID: \"085b04bc-51a6-4928-815d-f38fe27aea3d\") " pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.123293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.123248 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" Nov 01 05:14:11.496132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.496109 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-847cf6dc9d-lk8pc"] Nov 01 05:14:11.498504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:14:11.498477 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod085b04bc_51a6_4928_815d_f38fe27aea3d.slice/crio-0a1d107f5a12b03fab3298c9721a11aa9e90750346adef58e406976f8eb942e5 WatchSource:0}: Error finding container 0a1d107f5a12b03fab3298c9721a11aa9e90750346adef58e406976f8eb942e5: Status 404 returned error can't find the container with id 0a1d107f5a12b03fab3298c9721a11aa9e90750346adef58e406976f8eb942e5 Nov 01 05:14:11.902445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:11.902412 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" event={"ID":"085b04bc-51a6-4928-815d-f38fe27aea3d","Type":"ContainerStarted","Data":"0a1d107f5a12b03fab3298c9721a11aa9e90750346adef58e406976f8eb942e5"} Nov 01 05:14:13.913531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:13.913499 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" event={"ID":"085b04bc-51a6-4928-815d-f38fe27aea3d","Type":"ContainerStarted","Data":"8a4e0f7ab44f62987c8c9b55570aa5280e1460d2b950352823e390d30264c15a"} Nov 01 05:14:14.031466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.031415 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-847cf6dc9d-lk8pc" podStartSLOduration=2.3169729820000002 podStartE2EDuration="4.031400065s" totalImagesPullingTime="1.714427083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:14:14.030546167 +0000 UTC m=+669.171135089" watchObservedRunningTime="2026-11-01 05:14:14.031400065 +0000 UTC m=+669.171988984" Nov 01 05:14:14.084454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.084417 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:14:14.084683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.084651 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6547cd65c-7v7ck" podUID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" containerName="authorino" containerID="cri-o://b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11" gracePeriod=30 Nov 01 05:14:14.346340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.346310 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:14:14.463333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.462693 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\" (UniqueName: \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\") pod \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " Nov 01 05:14:14.463333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.462748 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\" (UniqueName: \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\") pod \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\" (UID: \"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978\") " Nov 01 05:14:14.472922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.471215 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2" pod "f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" (UID: "f4d8748e-98a9-4e85-a6b5-1ba7d9f02978"). InnerVolumeSpecName "kube-api-access-nl2v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:14:14.479310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.479239 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert" pod "f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" (UID: "f4d8748e-98a9-4e85-a6b5-1ba7d9f02978"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:14:14.563941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.563900 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2v2\" (UniqueName: \"kubernetes.io/projected/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-kube-api-access-nl2v2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:14:14.563941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.563943 4425 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978-tls-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:14:14.920528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.920495 4425 generic.go:356] "Generic (PLEG): container finished" podID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" containerID="b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11" exitCode=0 Nov 01 05:14:14.920974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.920555 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6547cd65c-7v7ck" Nov 01 05:14:14.920974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.920576 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6547cd65c-7v7ck" event={"ID":"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978","Type":"ContainerDied","Data":"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11"} Nov 01 05:14:14.920974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.920622 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6547cd65c-7v7ck" event={"ID":"f4d8748e-98a9-4e85-a6b5-1ba7d9f02978","Type":"ContainerDied","Data":"4bfdd6c3df5514cc8c142c3da7aa3f1ceedb1d07249fb0d46dffe7e0b2935ec5"} Nov 01 05:14:14.920974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.920640 4425 scope.go:122] "RemoveContainer" containerID="b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11" Nov 01 05:14:14.929710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.929687 4425 scope.go:122] "RemoveContainer" containerID="b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11" Nov 01 05:14:14.930058 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:14:14.930028 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11\": container with ID starting with b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11 not found: ID does not exist" containerID="b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11" Nov 01 05:14:14.930167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:14.930067 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11"} err="failed to get container status \"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11\": rpc error: code = NotFound desc = could not find container \"b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11\": container with ID starting with b7a80b697c8e3206388c757f6281ab8188f7ace5ca5c0d471afbb37f63d4eb11 not found: ID does not exist" Nov 01 05:14:15.087499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:15.087464 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:14:15.091171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:15.091147 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6547cd65c-7v7ck"] Nov 01 05:14:16.972600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:16.972527 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" path="/var/lib/kubelet/pods/f4d8748e-98a9-4e85-a6b5-1ba7d9f02978/volumes" Nov 01 05:14:26.948044 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:14:26.948004 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:14:51.232447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.232416 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:14:51.232916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.232822 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" containerName="authorino" Nov 01 05:14:51.232916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.232833 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" containerName="authorino" Nov 01 05:14:51.232916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.232913 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d8748e-98a9-4e85-a6b5-1ba7d9f02978" containerName="authorino" Nov 01 05:14:51.236124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.236107 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.289309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.289281 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\"" Nov 01 05:14:51.343374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.343340 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:14:51.345725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.345693 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.345868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.345763 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmlj\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.345868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.345847 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.345995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.345881 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.345995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.345906 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.346109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.346015 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.346109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.346046 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.346215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.346134 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.346289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.346230 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.446994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.446961 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447014 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447063 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447145 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmlj\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447186 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447209 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447233 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447295 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447324 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447645 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447732 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447755 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.447847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447818 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.448205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.447850 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.449856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.449831 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.449949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.449841 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.509239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.509171 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.509382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.509333 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmlj\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gtrfd\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:51.548163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:51.548133 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:52.005318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.005201 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:14:52.006303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:14:52.006277 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62701486_39cc_4ea4_bbd3_5c33dd61e773.slice/crio-6717ea568623b8f504b31b6d8c1ea3b5364baff152b91fb40ccc70a8563bd757 WatchSource:0}: Error finding container 6717ea568623b8f504b31b6d8c1ea3b5364baff152b91fb40ccc70a8563bd757: Status 404 returned error can't find the container with id 6717ea568623b8f504b31b6d8c1ea3b5364baff152b91fb40ccc70a8563bd757 Nov 01 05:14:52.009669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.009592 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:14:52.009801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.009679 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:14:52.009801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.009731 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:14:52.101693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.101655 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerStarted","Data":"ef7d4c7d9184114f7b8f79204b0f72039ee0ba85c205e6fd1353b62223d04ab7"} Nov 01 05:14:52.101880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.101722 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerStarted","Data":"6717ea568623b8f504b31b6d8c1ea3b5364baff152b91fb40ccc70a8563bd757"} Nov 01 05:14:52.225895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.225836 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podStartSLOduration=1.225821685 podStartE2EDuration="1.225821685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:14:52.223482558 +0000 UTC m=+707.364071509" watchObservedRunningTime="2026-11-01 05:14:52.225821685 +0000 UTC m=+707.366410604" Nov 01 05:14:52.548732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:52.548696 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:53.554118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:53.554093 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:54.116721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:54.116690 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:54.117682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:54.117660 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:14:56.945700 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:14:56.945662 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:14:59.186931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.186892 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 05:14:59.191469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.191445 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.222041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.222012 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.222177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.222047 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxz8\" (UniqueName: \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.245948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.245913 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a48099\"" Nov 01 05:14:59.313275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.301612 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a48099-dockercfg-zfhdp\"" Nov 01 05:14:59.313275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.304516 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 05:14:59.328124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.328084 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.329324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.329302 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxz8\" (UniqueName: \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.329889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.329221 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.354574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.354541 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 05:14:59.358220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.358200 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.391413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.391386 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxz8\" (UniqueName: \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\") pod \"payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.396384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.396357 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 05:14:59.430514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.430478 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.430660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.430618 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shp2d\" (UniqueName: \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.505308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.505174 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:14:59.531854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.531819 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shp2d\" (UniqueName: \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.531999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.531962 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.532673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.532651 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.593800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.593770 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shp2d\" (UniqueName: \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\") pod \"payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.672025 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.671990 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:14:59.876849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:14:59.876826 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 05:14:59.878522 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:14:59.878496 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06eda37_d96c_4770_a9a2_80ad231053a9.slice/crio-dcca352d4ff1b9280d59beaa7c96e6147b16f25efe14d264f2fa8bae1ee7c99b WatchSource:0}: Error finding container dcca352d4ff1b9280d59beaa7c96e6147b16f25efe14d264f2fa8bae1ee7c99b: Status 404 returned error can't find the container with id dcca352d4ff1b9280d59beaa7c96e6147b16f25efe14d264f2fa8bae1ee7c99b Nov 01 05:15:00.047192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.047164 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 05:15:00.049107 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:00.049071 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907e94f8_1acf_4653_9d52_d6af35d9097c.slice/crio-ffc0af56e99b05a1e90feef14459fe5992f44f5ecc236a8863523b405747c2eb WatchSource:0}: Error finding container ffc0af56e99b05a1e90feef14459fe5992f44f5ecc236a8863523b405747c2eb: Status 404 returned error can't find the container with id ffc0af56e99b05a1e90feef14459fe5992f44f5ecc236a8863523b405747c2eb Nov 01 05:15:00.149987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.149892 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" event={"ID":"907e94f8-1acf-4653-9d52-d6af35d9097c","Type":"ContainerStarted","Data":"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465"} Nov 01 05:15:00.149987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.149952 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" event={"ID":"907e94f8-1acf-4653-9d52-d6af35d9097c","Type":"ContainerStarted","Data":"ffc0af56e99b05a1e90feef14459fe5992f44f5ecc236a8863523b405747c2eb"} Nov 01 05:15:00.150195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.150050 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:15:00.151491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.151468 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" event={"ID":"d06eda37-d96c-4770-a9a2-80ad231053a9","Type":"ContainerStarted","Data":"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6"} Nov 01 05:15:00.151598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.151506 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" event={"ID":"d06eda37-d96c-4770-a9a2-80ad231053a9","Type":"ContainerStarted","Data":"dcca352d4ff1b9280d59beaa7c96e6147b16f25efe14d264f2fa8bae1ee7c99b"} Nov 01 05:15:00.151598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.151534 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:15:00.267733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.267664 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" podStartSLOduration=1.267642867 podStartE2EDuration="1.267642867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:00.267493288 +0000 UTC m=+715.408082241" watchObservedRunningTime="2026-11-01 05:15:00.267642867 +0000 UTC m=+715.408231787" Nov 01 05:15:00.386967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:00.386904 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" podStartSLOduration=1.386884492 podStartE2EDuration="1.386884492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:00.385841877 +0000 UTC m=+715.526430822" watchObservedRunningTime="2026-11-01 05:15:00.386884492 +0000 UTC m=+715.527473415" Nov 01 05:15:04.062916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.062876 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 05:15:04.066811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.066786 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.120127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.120092 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7d1ec3\"" Nov 01 05:15:04.173750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.172069 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7d1ec3-dockercfg-xjqn9\"" Nov 01 05:15:04.177763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.176835 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 05:15:04.178996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.178965 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlh6\" (UniqueName: \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.179357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.179326 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.223632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.223602 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 05:15:04.227215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.227199 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.266727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.266692 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 05:15:04.280605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.280572 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.280747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.280651 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4r5\" (UniqueName: \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.280747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.280735 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlh6\" (UniqueName: \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.280834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.280780 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.281752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.281730 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.345959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.345889 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlh6\" (UniqueName: \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\") pod \"payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.381074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.381044 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:04.381436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.381413 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.381486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.381462 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4r5\" (UniqueName: \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.382059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.382041 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.447377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.447343 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4r5\" (UniqueName: \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\") pod \"payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.540213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.540175 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:04.754996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.754966 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 05:15:04.756587 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:04.756560 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6faddc_9c67_4a16_b812_2c69a3c82886.slice/crio-ec3c32be6ff817f0bfc011937d527affab76ddbb697bc0ff6553f4cd16747d3c WatchSource:0}: Error finding container ec3c32be6ff817f0bfc011937d527affab76ddbb697bc0ff6553f4cd16747d3c: Status 404 returned error can't find the container with id ec3c32be6ff817f0bfc011937d527affab76ddbb697bc0ff6553f4cd16747d3c Nov 01 05:15:04.921492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:04.921464 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 05:15:04.923209 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:04.923188 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb3bd52_fca2_41b1_9503_ad8e75da3e0f.slice/crio-faff8f113e800cb40d06c5fb5363136ef8fce33f5e653334bf7f26d2118798cf WatchSource:0}: Error finding container faff8f113e800cb40d06c5fb5363136ef8fce33f5e653334bf7f26d2118798cf: Status 404 returned error can't find the container with id faff8f113e800cb40d06c5fb5363136ef8fce33f5e653334bf7f26d2118798cf Nov 01 05:15:05.184244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.184156 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" event={"ID":"be6faddc-9c67-4a16-b812-2c69a3c82886","Type":"ContainerStarted","Data":"aef9e2a1fa580b23f5d6d073272a1cfff34f2b73bc0d96e0debc3f5928dc794e"} Nov 01 05:15:05.184244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.184246 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" event={"ID":"be6faddc-9c67-4a16-b812-2c69a3c82886","Type":"ContainerStarted","Data":"ec3c32be6ff817f0bfc011937d527affab76ddbb697bc0ff6553f4cd16747d3c"} Nov 01 05:15:05.184723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.184295 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:05.185889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.185865 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" event={"ID":"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f","Type":"ContainerStarted","Data":"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5"} Nov 01 05:15:05.186024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.185908 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" event={"ID":"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f","Type":"ContainerStarted","Data":"faff8f113e800cb40d06c5fb5363136ef8fce33f5e653334bf7f26d2118798cf"} Nov 01 05:15:05.186024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.185968 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:05.302171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.302119 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" podStartSLOduration=1.3021064949999999 podStartE2EDuration="1.302106495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:05.301334013 +0000 UTC m=+720.441922953" watchObservedRunningTime="2026-11-01 05:15:05.302106495 +0000 UTC m=+720.442695414" Nov 01 05:15:05.422028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:05.421974 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" podStartSLOduration=1.421957463 podStartE2EDuration="1.421957463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:05.420346087 +0000 UTC m=+720.560935025" watchObservedRunningTime="2026-11-01 05:15:05.421957463 +0000 UTC m=+720.562546411" Nov 01 05:15:06.572155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.572116 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:15:06.577115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.577088 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.607062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.607020 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.607181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.607168 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxspb\" (UniqueName: \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.684519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.684476 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3d70ff\"" Nov 01 05:15:06.684519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.684476 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3d70ff-dockercfg-m96mr\"" Nov 01 05:15:06.687779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.687749 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:15:06.708664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.708637 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxspb\" (UniqueName: \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.708802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.708723 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.709518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.709497 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.715291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.715268 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:15:06.719089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.719069 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.771480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.771447 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxspb\" (UniqueName: \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\") pod \"payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.777712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.777683 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:15:06.809466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.809437 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndpd\" (UniqueName: \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.809632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.809493 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.891353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.891245 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:06.910775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.910746 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndpd\" (UniqueName: \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.910928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.910800 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.911434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.911412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:06.973768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:06.973729 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndpd\" (UniqueName: \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\") pod \"payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:07.035879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:07.035842 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:07.264394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:07.264369 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:15:07.266116 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:07.266083 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce603a8_4361_4c41_b09a_04d84a69b5d5.slice/crio-0f5c89d7f25bffcf057581add0ba2f7e670dfd16bfdbab508ad6c44822385266 WatchSource:0}: Error finding container 0f5c89d7f25bffcf057581add0ba2f7e670dfd16bfdbab508ad6c44822385266: Status 404 returned error can't find the container with id 0f5c89d7f25bffcf057581add0ba2f7e670dfd16bfdbab508ad6c44822385266 Nov 01 05:15:07.412231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:07.412206 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:15:07.413753 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:07.413725 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea92693_1c83_4e04_a4aa_7d0a1ab69159.slice/crio-38dfdd831462c880ed7d311f073d42d41780f7771fb55eb4438d4c98ec980238 WatchSource:0}: Error finding container 38dfdd831462c880ed7d311f073d42d41780f7771fb55eb4438d4c98ec980238: Status 404 returned error can't find the container with id 38dfdd831462c880ed7d311f073d42d41780f7771fb55eb4438d4c98ec980238 Nov 01 05:15:08.207949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.207905 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" event={"ID":"8ce603a8-4361-4c41-b09a-04d84a69b5d5","Type":"ContainerStarted","Data":"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9"} Nov 01 05:15:08.208413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.207999 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" event={"ID":"8ce603a8-4361-4c41-b09a-04d84a69b5d5","Type":"ContainerStarted","Data":"0f5c89d7f25bffcf057581add0ba2f7e670dfd16bfdbab508ad6c44822385266"} Nov 01 05:15:08.208413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.208207 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:08.209648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.209626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" event={"ID":"cea92693-1c83-4e04-a4aa-7d0a1ab69159","Type":"ContainerStarted","Data":"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016"} Nov 01 05:15:08.209737 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.209663 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" event={"ID":"cea92693-1c83-4e04-a4aa-7d0a1ab69159","Type":"ContainerStarted","Data":"38dfdd831462c880ed7d311f073d42d41780f7771fb55eb4438d4c98ec980238"} Nov 01 05:15:08.209737 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.209683 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:08.329785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.329732 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" podStartSLOduration=2.329718157 podStartE2EDuration="2.329718157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:08.327973343 +0000 UTC m=+723.468562262" watchObservedRunningTime="2026-11-01 05:15:08.329718157 +0000 UTC m=+723.470307075" Nov 01 05:15:08.487318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:08.487213 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" podStartSLOduration=2.487199931 podStartE2EDuration="2.487199931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:08.485441874 +0000 UTC m=+723.626030798" watchObservedRunningTime="2026-11-01 05:15:08.487199931 +0000 UTC m=+723.627788850" Nov 01 05:15:11.159891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:11.159864 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 05:15:11.160298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:11.159941 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 05:15:16.194593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:16.194567 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 05:15:16.194989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:16.194633 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 05:15:19.217851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.217816 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:15:19.218286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.218074 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:15:19.494717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.494635 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55"] Nov 01 05:15:19.498308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.498282 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.557769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.557738 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:15:19.557769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.557753 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:15:19.558000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.557794 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:15:19.558167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.558145 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:15:19.560905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.560883 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55"] Nov 01 05:15:19.638089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.638052 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndc55\" (UniqueName: \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.638296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.638119 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.638296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.638207 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.739021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.738979 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.739201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.739031 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.739201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.739102 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndc55\" (UniqueName: \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.739679 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.739646 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.741577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.741556 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.802410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.802342 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndc55\" (UniqueName: \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\") pod \"maas-api-revoke-sub-6c9a82e8-s7m55\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:19.813029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:19.813006 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:20.187734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:20.187673 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55"] Nov 01 05:15:20.190034 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:20.190004 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31165c24_7a04_4db2_8939_943577098990.slice/crio-129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f WatchSource:0}: Error finding container 129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f: Status 404 returned error can't find the container with id 129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f Nov 01 05:15:20.285261 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:20.285229 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" event={"ID":"31165c24-7a04-4db2-8939-943577098990","Type":"ContainerStarted","Data":"129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f"} Nov 01 05:15:26.387598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:26.387560 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" event={"ID":"31165c24-7a04-4db2-8939-943577098990","Type":"ContainerStarted","Data":"5df4e93c62052f9d0853a5f758d553c603ff6e01c8fc1fb1975b81591b003b8d"} Nov 01 05:15:26.568898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:26.568832 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" podStartSLOduration=2.1820666109999998 podStartE2EDuration="7.568812079s" totalImagesPullingTime="5.386745468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:26.504360554 +0000 UTC m=+741.644949493" watchObservedRunningTime="2026-11-01 05:15:26.568812079 +0000 UTC m=+741.709401001" Nov 01 05:15:26.947208 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:15:26.947170 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:15:27.395234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:27.395193 4425 generic.go:356] "Generic (PLEG): container finished" podID="31165c24-7a04-4db2-8939-943577098990" containerID="5df4e93c62052f9d0853a5f758d553c603ff6e01c8fc1fb1975b81591b003b8d" exitCode=0 Nov 01 05:15:27.395646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:27.395236 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" event={"ID":"31165c24-7a04-4db2-8939-943577098990","Type":"ContainerDied","Data":"5df4e93c62052f9d0853a5f758d553c603ff6e01c8fc1fb1975b81591b003b8d"} Nov 01 05:15:28.550411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.550382 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:28.614858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.614830 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\" (UniqueName: \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\") pod \"31165c24-7a04-4db2-8939-943577098990\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " Nov 01 05:15:28.615035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.614969 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\") pod \"31165c24-7a04-4db2-8939-943577098990\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " Nov 01 05:15:28.615035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.615013 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\") pod \"31165c24-7a04-4db2-8939-943577098990\" (UID: \"31165c24-7a04-4db2-8939-943577098990\") " Nov 01 05:15:28.615368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.615341 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca" pod "31165c24-7a04-4db2-8939-943577098990" (UID: "31165c24-7a04-4db2-8939-943577098990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:15:28.617187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.617159 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55" pod "31165c24-7a04-4db2-8939-943577098990" (UID: "31165c24-7a04-4db2-8939-943577098990"). InnerVolumeSpecName "kube-api-access-ndc55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:15:28.617313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.617238 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca" pod "31165c24-7a04-4db2-8939-943577098990" (UID: "31165c24-7a04-4db2-8939-943577098990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:15:28.716270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.716170 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31165c24-7a04-4db2-8939-943577098990-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:15:28.716270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.716213 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31165c24-7a04-4db2-8939-943577098990-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:15:28.716270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:28.716228 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndc55\" (UniqueName: \"kubernetes.io/projected/31165c24-7a04-4db2-8939-943577098990-kube-api-access-ndc55\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:15:29.405998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:29.405965 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" Nov 01 05:15:29.405998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:29.405978 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55" event={"ID":"31165c24-7a04-4db2-8939-943577098990","Type":"ContainerDied","Data":"129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f"} Nov 01 05:15:29.406187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:29.406033 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="129dac9d40b3ae93c5f1add847f65e600fbc8561ea6000b0b6e27c3386661b1f" Nov 01 05:15:30.412973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.412938 4425 generic.go:356] "Generic (PLEG): container finished" podID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerID="ef7d4c7d9184114f7b8f79204b0f72039ee0ba85c205e6fd1353b62223d04ab7" exitCode=0 Nov 01 05:15:30.413590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.412995 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerDied","Data":"ef7d4c7d9184114f7b8f79204b0f72039ee0ba85c205e6fd1353b62223d04ab7"} Nov 01 05:15:30.413590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.413481 4425 scope.go:122] "RemoveContainer" containerID="ef7d4c7d9184114f7b8f79204b0f72039ee0ba85c205e6fd1353b62223d04ab7" Nov 01 05:15:30.414765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.414732 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:30.414849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.414807 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:30.414849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:30.414841 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:31.420307 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:31.420278 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerStarted","Data":"fca2cff5515d1aad0326356dabc6281c01c1759da88ff03227133f48a8ec1614"} Nov 01 05:15:31.550285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:31.548649 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:15:31.550285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:31.548764 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 05:15:31.550285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:31.548808 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 05:15:32.556734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:32.556696 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 05:15:32.557393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:32.557358 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 05:15:33.548798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:33.548757 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 05:15:33.549109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:33.549057 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 05:15:34.566427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.566394 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 05:15:34.566911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.566457 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 05:15:34.873218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.873144 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 05:15:34.873597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.873584 4425 cpu_manager.go:414] "containerMap: removing container" podUID="31165c24-7a04-4db2-8939-943577098990" containerName="revoke-keys" Nov 01 05:15:34.873640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.873599 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31165c24-7a04-4db2-8939-943577098990" containerName="revoke-keys" Nov 01 05:15:34.873702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.873694 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="31165c24-7a04-4db2-8939-943577098990" containerName="revoke-keys" Nov 01 05:15:34.880102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.880082 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:34.937469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.937440 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 05:15:34.984076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.984027 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"default-dockercfg-4l4n6\"" Nov 01 05:15:34.984076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.984067 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"simulator-w0-7d1ec3-kserve-self-signed-certs\"" Nov 01 05:15:34.985505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.985490 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"openshift-service-ca.crt\"" Nov 01 05:15:34.985578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:34.985493 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"kube-root-ca.crt\"" Nov 01 05:15:35.005012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.004984 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbxk\" (UniqueName: \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.005081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.005020 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.005081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.005041 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.005181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.005165 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.005244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.005204 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.005410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.005396 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106024 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbxk\" (UniqueName: \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106064 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106082 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106120 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106231 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106386 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106554 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106659 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.106869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.106851 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.108732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.108711 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.109039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.109016 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.169164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.169075 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbxk\" (UniqueName: \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\") pod \"simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.192625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.192593 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:35.566338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.560706 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 05:15:35.566338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.560762 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 05:15:35.590912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:35.590887 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 05:15:35.592968 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:35.592937 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a81a698_5cd0_4d60_9e57_71b6036dd7c0.slice/crio-129e0730f9a01ce2d618683233ce70e2f5f17079c6613fd9075af9a189262695 WatchSource:0}: Error finding container 129e0730f9a01ce2d618683233ce70e2f5f17079c6613fd9075af9a189262695: Status 404 returned error can't find the container with id 129e0730f9a01ce2d618683233ce70e2f5f17079c6613fd9075af9a189262695 Nov 01 05:15:36.446907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.446864 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" event={"ID":"7a81a698-5cd0-4d60-9e57-71b6036dd7c0","Type":"ContainerStarted","Data":"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb"} Nov 01 05:15:36.446907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.446909 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" event={"ID":"7a81a698-5cd0-4d60-9e57-71b6036dd7c0","Type":"ContainerStarted","Data":"129e0730f9a01ce2d618683233ce70e2f5f17079c6613fd9075af9a189262695"} Nov 01 05:15:36.566461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.566405 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" podStartSLOduration=2.566388658 podStartE2EDuration="2.566388658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:36.565078363 +0000 UTC m=+751.705667313" watchObservedRunningTime="2026-11-01 05:15:36.566388658 +0000 UTC m=+751.706977583" Nov 01 05:15:36.796044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.795945 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:15:36.796455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.796414 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:15:36.797575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:36.797554 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:15:37.193525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:37.193488 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:45.193436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:45.193399 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:45.206057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:45.206031 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:45.511283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:45.511191 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 05:15:48.557212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.557134 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:15:48.562943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.562922 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.616699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.616672 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ea125-openshift-default-dockercfg-6b9j7\"" Nov 01 05:15:48.651899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.651860 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8lj\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.651904 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.651930 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652066 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652101 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652155 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652234 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652301 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.652344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.652325 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.669435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.669404 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:15:48.753499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753461 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8lj\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753508 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753534 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753559 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753588 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753734 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753845 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753871 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.753908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753889 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.754148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.753971 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.754148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.754060 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.754148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.754105 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.754307 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.754284 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.754514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.754498 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.756754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.756722 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.756858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.756842 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.815725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.815684 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.815870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.815855 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8lj\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\") pod \"e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:48.876378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:48.876344 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:49.309323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.309203 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:15:49.345718 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:49.345691 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2276e4f5_451a_4144_90af_e4f5f1eb5748.slice/crio-11116d9da5fb0519ede9c7aec7c83d5750d6a183713c07aa6d40a370e149fcc6 WatchSource:0}: Error finding container 11116d9da5fb0519ede9c7aec7c83d5750d6a183713c07aa6d40a370e149fcc6: Status 404 returned error can't find the container with id 11116d9da5fb0519ede9c7aec7c83d5750d6a183713c07aa6d40a370e149fcc6 Nov 01 05:15:49.348074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.348038 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:49.348167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.348108 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:49.348167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.348137 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:15:49.522527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.522491 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerStarted","Data":"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c"} Nov 01 05:15:49.522729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.522558 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerStarted","Data":"11116d9da5fb0519ede9c7aec7c83d5750d6a183713c07aa6d40a370e149fcc6"} Nov 01 05:15:49.644976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.644915 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podStartSLOduration=1.644896175 podStartE2EDuration="1.644896175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:49.644170974 +0000 UTC m=+764.784759894" watchObservedRunningTime="2026-11-01 05:15:49.644896175 +0000 UTC m=+764.785485096" Nov 01 05:15:49.877093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:49.877059 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:50.881873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:50.881839 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:51.533006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:51.532970 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:51.533893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:51.533873 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:15:56.501068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.501025 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:15:56.504749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.504732 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.534097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.534070 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.534221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.534128 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhg8\" (UniqueName: \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.559683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.559644 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ea125\"" Nov 01 05:15:56.610336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.610310 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ea125-dockercfg-d85wp\"" Nov 01 05:15:56.612540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.612512 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:15:56.634736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.634710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.634887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.634768 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhg8\" (UniqueName: \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.635514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.635496 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.696864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.696836 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhg8\" (UniqueName: \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\") pod \"payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.819047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:56.819023 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:56.947739 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:15:56.947700 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:15:57.195106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:57.195078 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:15:57.196568 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:15:57.196542 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999804dc_8517_448d_a6bd_6528f0b6fd84.slice/crio-542dbfc4caa04f7f54412283f8c9cc0a007cf0d57837b722fa976e451ffc5d4d WatchSource:0}: Error finding container 542dbfc4caa04f7f54412283f8c9cc0a007cf0d57837b722fa976e451ffc5d4d: Status 404 returned error can't find the container with id 542dbfc4caa04f7f54412283f8c9cc0a007cf0d57837b722fa976e451ffc5d4d Nov 01 05:15:57.564248 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:57.564219 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" event={"ID":"999804dc-8517-448d-a6bd-6528f0b6fd84","Type":"ContainerStarted","Data":"938d2c016ceb1c28dc11da8c826c24fefcffecbc49d669c21a66571aadd9305c"} Nov 01 05:15:57.564650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:57.564278 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" event={"ID":"999804dc-8517-448d-a6bd-6528f0b6fd84","Type":"ContainerStarted","Data":"542dbfc4caa04f7f54412283f8c9cc0a007cf0d57837b722fa976e451ffc5d4d"} Nov 01 05:15:57.564650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:57.564309 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:15:57.683133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:15:57.683079 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" podStartSLOduration=1.6830650600000001 podStartE2EDuration="1.68306506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:57.681415887 +0000 UTC m=+772.822004828" watchObservedRunningTime="2026-11-01 05:15:57.68306506 +0000 UTC m=+772.823653979" Nov 01 05:16:08.572541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:08.572514 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:16:17.026542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.026507 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 05:16:17.031681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.031659 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.136739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.136706 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"premium-w0-7d1ec3-kserve-self-signed-certs\"" Nov 01 05:16:17.139480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.139456 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 05:16:17.147197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147172 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.147324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.147431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147414 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.147483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147469 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.147526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147514 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdl2\" (UniqueName: \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.147590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.147574 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248297 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248359 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248387 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdl2\" (UniqueName: \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248488 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248549 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248584 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248801 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.248967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248928 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.249031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.248981 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.250774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.250755 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.251076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.251058 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.311296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.311268 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdl2\" (UniqueName: \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\") pod \"premium-w0-7d1ec3-kserve-54777b9cff-c9hpv\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.343484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.343445 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:17.725727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:17.725703 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 05:16:17.727319 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:16:17.727286 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ded2e64_9d29_4f60_bf61_b99176c5721d.slice/crio-1e1b426c344afb48c70cc2adca6617f28782c64d27a699a1eddb02feefb7b2a9 WatchSource:0}: Error finding container 1e1b426c344afb48c70cc2adca6617f28782c64d27a699a1eddb02feefb7b2a9: Status 404 returned error can't find the container with id 1e1b426c344afb48c70cc2adca6617f28782c64d27a699a1eddb02feefb7b2a9 Nov 01 05:16:18.675964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:18.675918 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" event={"ID":"7ded2e64-9d29-4f60-bf61-b99176c5721d","Type":"ContainerStarted","Data":"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6"} Nov 01 05:16:18.676409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:18.675977 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" event={"ID":"7ded2e64-9d29-4f60-bf61-b99176c5721d","Type":"ContainerStarted","Data":"1e1b426c344afb48c70cc2adca6617f28782c64d27a699a1eddb02feefb7b2a9"} Nov 01 05:16:18.802779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:18.802721 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podStartSLOduration=2.802703858 podStartE2EDuration="2.802703858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:18.802207425 +0000 UTC m=+793.942796345" watchObservedRunningTime="2026-11-01 05:16:18.802703858 +0000 UTC m=+793.943292776" Nov 01 05:16:19.343605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:19.343570 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:26.971140 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:16:26.971090 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:16:27.344535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:27.344500 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:27.358667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:27.358636 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:27.750303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:27.750198 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 05:16:30.753885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.753786 4425 generic.go:356] "Generic (PLEG): container finished" podID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerID="9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c" exitCode=0 Nov 01 05:16:30.753885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.753847 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerDied","Data":"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c"} Nov 01 05:16:30.754476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.754434 4425 scope.go:122] "RemoveContainer" containerID="9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c" Nov 01 05:16:30.756100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.756071 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:30.756200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.756139 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:30.756200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:30.756172 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:31.772109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:31.772075 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerStarted","Data":"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465"} Nov 01 05:16:31.876785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:31.876756 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:16:31.876987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:31.876969 4425 patch_prober.go:29] interesting pod/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 05:16:31.877034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:31.877014 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 05:16:32.877027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:32.876987 4425 patch_prober.go:29] interesting pod/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 05:16:32.877489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:32.877053 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 05:16:33.877274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:33.877087 4425 patch_prober.go:29] interesting pod/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 05:16:33.877274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:33.877158 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 05:16:34.876983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:34.876944 4425 patch_prober.go:29] interesting pod/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 05:16:34.877185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:34.877006 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 05:16:35.877690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:35.877650 4425 patch_prober.go:29] interesting pod/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 05:16:35.878121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:35.877704 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 05:16:36.533480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:36.533443 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:16:36.880813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:36.880787 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:16:36.881731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:36.881706 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:16:44.061413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.061380 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:16:44.065401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.065374 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.167710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.167677 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:16:44.176021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.175990 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176073 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176107 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176131 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176361 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176416 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176467 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176454 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176563 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxrr\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.176621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.176606 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.215185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.215151 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-af8051-openshift-default-dockercfg-h7cwz\"" Nov 01 05:16:44.278225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278186 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278235 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278292 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278426 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxrr\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278474 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278571 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278626 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278655 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278680 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278690 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278738 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278754 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.278971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.278934 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.279534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.279427 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.281008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.280981 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.281209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.281192 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.340625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.340546 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.340776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.340718 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxrr\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\") pod \"e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.378169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.378137 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:44.807468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.807444 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:16:44.808882 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:16:44.808853 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9f4419_e6d4_41aa_8154_f24d3d8aa2ff.slice/crio-9a62b035d055ec4d17da2820c1ee5dc8f4060dd34d88edce1e7b8c12f6fb18fd WatchSource:0}: Error finding container 9a62b035d055ec4d17da2820c1ee5dc8f4060dd34d88edce1e7b8c12f6fb18fd: Status 404 returned error can't find the container with id 9a62b035d055ec4d17da2820c1ee5dc8f4060dd34d88edce1e7b8c12f6fb18fd Nov 01 05:16:44.811546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.811516 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:44.811629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.811579 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:44.811629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.811604 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:16:44.849766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:44.849728 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerStarted","Data":"9a62b035d055ec4d17da2820c1ee5dc8f4060dd34d88edce1e7b8c12f6fb18fd"} Nov 01 05:16:45.856229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:45.856187 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerStarted","Data":"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5"} Nov 01 05:16:45.979809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:45.979751 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podStartSLOduration=1.979737735 podStartE2EDuration="1.979737735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:45.978151919 +0000 UTC m=+821.118740863" watchObservedRunningTime="2026-11-01 05:16:45.979737735 +0000 UTC m=+821.120326654" Nov 01 05:16:46.379111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:46.379077 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:46.384471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:46.384441 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:46.861792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:46.861757 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:46.862740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:46.862721 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:16:52.767190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.767154 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:16:52.771299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.771274 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:52.828321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.828286 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:16:52.876035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.876007 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af8051\"" Nov 01 05:16:52.876213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.876106 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af8051-dockercfg-jpmwv\"" Nov 01 05:16:52.878640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.878610 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:52.878789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.878657 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8v4\" (UniqueName: \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:52.980763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.980717 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:52.980936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.980820 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8v4\" (UniqueName: \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:52.981500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:52.981480 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:53.043408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.043327 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8v4\" (UniqueName: \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\") pod \"payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:53.086948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.086908 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:53.471744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.471712 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:16:53.471882 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:16:53.471720 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5359b1a_cce7_4a07_be2d_c3a6a9801891.slice/crio-7effcdd1b9213d19b0f2fbc257c0b8fa4df70e9afe90df01e962d7d02c92d0f8 WatchSource:0}: Error finding container 7effcdd1b9213d19b0f2fbc257c0b8fa4df70e9afe90df01e962d7d02c92d0f8: Status 404 returned error can't find the container with id 7effcdd1b9213d19b0f2fbc257c0b8fa4df70e9afe90df01e962d7d02c92d0f8 Nov 01 05:16:53.904420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.904387 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" event={"ID":"f5359b1a-cce7-4a07-be2d-c3a6a9801891","Type":"ContainerStarted","Data":"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7"} Nov 01 05:16:53.904916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.904442 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" event={"ID":"f5359b1a-cce7-4a07-be2d-c3a6a9801891","Type":"ContainerStarted","Data":"7effcdd1b9213d19b0f2fbc257c0b8fa4df70e9afe90df01e962d7d02c92d0f8"} Nov 01 05:16:53.904916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:53.904602 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:16:54.024554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:16:54.024490 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" podStartSLOduration=2.024469512 podStartE2EDuration="2.024469512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:54.022346081 +0000 UTC m=+829.162935001" watchObservedRunningTime="2026-11-01 05:16:54.024469512 +0000 UTC m=+829.165058434" Nov 01 05:16:56.947076 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:16:56.947036 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:17:04.914594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:04.914545 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:17:26.974681 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:17:26.974579 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:17:28.486981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.486941 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 05:17:28.491104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.491088 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.601458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.601406 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnndl\" (UniqueName: \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.602149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.602122 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.612540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.612508 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a12c35\"" Nov 01 05:17:28.615204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.612528 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a12c35-dockercfg-kjbwt\"" Nov 01 05:17:28.616726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.616700 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 05:17:28.668450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.668405 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 05:17:28.673648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.673615 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.700223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.700184 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 05:17:28.703985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.703959 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.704121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.704023 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnndl\" (UniqueName: \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.704683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.704658 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.773990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.773913 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnndl\" (UniqueName: \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\") pod \"payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.804391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.804360 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:28.804834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.804810 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkhp\" (UniqueName: \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.804941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.804859 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.906081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.906053 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkhp\" (UniqueName: \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.906197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.906095 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.906854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.906837 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.970131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.970090 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkhp\" (UniqueName: \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\") pod \"payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:28.987143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:28.987107 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:29.200118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:29.200093 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 05:17:29.201338 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:17:29.201311 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040645ce_dc42_4069_b610_9554f4444e52.slice/crio-154c77da1250985fc93df1ffb1ef7bc9d8fd435cb463756c9b10b42a7697d801 WatchSource:0}: Error finding container 154c77da1250985fc93df1ffb1ef7bc9d8fd435cb463756c9b10b42a7697d801: Status 404 returned error can't find the container with id 154c77da1250985fc93df1ffb1ef7bc9d8fd435cb463756c9b10b42a7697d801 Nov 01 05:17:29.368640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:29.368617 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 05:17:29.371374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:17:29.371340 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e904aa_5f45_4e1c_ab54_c2c8e6170958.slice/crio-00b32976e4b6ea1a382ad073d971877b20b410d5990eff9e78314ddb2fc78450 WatchSource:0}: Error finding container 00b32976e4b6ea1a382ad073d971877b20b410d5990eff9e78314ddb2fc78450: Status 404 returned error can't find the container with id 00b32976e4b6ea1a382ad073d971877b20b410d5990eff9e78314ddb2fc78450 Nov 01 05:17:30.105669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.105626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" event={"ID":"040645ce-dc42-4069-b610-9554f4444e52","Type":"ContainerStarted","Data":"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99"} Nov 01 05:17:30.106147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.105684 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" event={"ID":"040645ce-dc42-4069-b610-9554f4444e52","Type":"ContainerStarted","Data":"154c77da1250985fc93df1ffb1ef7bc9d8fd435cb463756c9b10b42a7697d801"} Nov 01 05:17:30.106147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.105825 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:30.107304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.107276 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" event={"ID":"90e904aa-5f45-4e1c-ab54-c2c8e6170958","Type":"ContainerStarted","Data":"74697569666c491e4fa7de7f50dfb3234a61a8661df0302f69ce857a763fc04f"} Nov 01 05:17:30.107434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.107310 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" event={"ID":"90e904aa-5f45-4e1c-ab54-c2c8e6170958","Type":"ContainerStarted","Data":"00b32976e4b6ea1a382ad073d971877b20b410d5990eff9e78314ddb2fc78450"} Nov 01 05:17:30.107434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.107385 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:30.231924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.231866 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" podStartSLOduration=2.231853084 podStartE2EDuration="2.231853084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:17:30.231312698 +0000 UTC m=+865.371901618" watchObservedRunningTime="2026-11-01 05:17:30.231853084 +0000 UTC m=+865.372442002" Nov 01 05:17:30.367343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:30.361784 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" podStartSLOduration=2.361761175 podStartE2EDuration="2.361761175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:17:30.359717432 +0000 UTC m=+865.500306352" watchObservedRunningTime="2026-11-01 05:17:30.361761175 +0000 UTC m=+865.502350097" Nov 01 05:17:32.903445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:32.903405 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:17:32.903883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:32.903478 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:17:33.276543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.276468 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr"] Nov 01 05:17:33.280530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.280513 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.355184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.355138 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.355403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.355206 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.355403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.355370 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkdp\" (UniqueName: \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.385532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.385503 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:17:33.385713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.385550 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:17:33.385713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.385552 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:17:33.385713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.385613 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:17:33.389799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.389775 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr"] Nov 01 05:17:33.456967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.456929 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.457177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.456977 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.457177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.457006 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkdp\" (UniqueName: \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.457820 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.457795 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.459678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.459638 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.519201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.519164 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkdp\" (UniqueName: \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\") pod \"maas-api-revoke-sub-c6f56600-ts9fr\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.594794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.594761 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:33.980189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:33.980158 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr"] Nov 01 05:17:33.982297 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:17:33.982232 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb1ef9_e05e_4d4f_84d7_8c281c39e3bf.slice/crio-43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a WatchSource:0}: Error finding container 43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a: Status 404 returned error can't find the container with id 43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a Nov 01 05:17:34.133673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:34.133638 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" event={"ID":"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf","Type":"ContainerStarted","Data":"43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a"} Nov 01 05:17:35.140828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.140795 4425 generic.go:356] "Generic (PLEG): container finished" podID="67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" containerID="cc1abb4545ff23a9691397a0b39b20dd694b9c088eb455c799ad3595aec71216" exitCode=0 Nov 01 05:17:35.141324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.140886 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" event={"ID":"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf","Type":"ContainerDied","Data":"cc1abb4545ff23a9691397a0b39b20dd694b9c088eb455c799ad3595aec71216"} Nov 01 05:17:35.142666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.142635 4425 generic.go:356] "Generic (PLEG): container finished" podID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerID="cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5" exitCode=0 Nov 01 05:17:35.142791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.142679 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerDied","Data":"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5"} Nov 01 05:17:35.143074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.143057 4425 scope.go:122] "RemoveContainer" containerID="cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5" Nov 01 05:17:35.144492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.144463 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:17:35.144577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.144528 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:17:35.144577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:35.144560 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:17:36.170550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.170511 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerStarted","Data":"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4"} Nov 01 05:17:36.334227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.334200 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:36.379125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.379085 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:17:36.379473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.379451 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:36.379532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.379503 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:36.492856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.492770 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\" (UniqueName: \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\") pod \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " Nov 01 05:17:36.492856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.492833 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\") pod \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " Nov 01 05:17:36.493098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.492968 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\") pod \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\" (UID: \"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf\") " Nov 01 05:17:36.493444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.493411 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca" pod "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" (UID: "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:17:36.495199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.495172 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca" pod "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" (UID: "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:17:36.495341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.495318 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp" pod "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" (UID: "67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf"). InnerVolumeSpecName "kube-api-access-fqkdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:17:36.594110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.594078 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:36.594110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.594108 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:36.594110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:36.594119 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkdp\" (UniqueName: \"kubernetes.io/projected/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf-kube-api-access-fqkdp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:37.177398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:37.177365 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" Nov 01 05:17:37.177907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:37.177394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr" event={"ID":"67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf","Type":"ContainerDied","Data":"43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a"} Nov 01 05:17:37.177907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:37.177455 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c592ea9772e2cc044b9d37b8ad021134888bf69dfd4ff7938563a5ec326c2a" Nov 01 05:17:37.378947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:37.378914 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:37.379129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:37.378967 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:38.379205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:38.379172 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:38.379668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:38.379238 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:39.379208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:39.379168 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:39.379743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:39.379285 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:40.378944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:40.378897 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:40.379161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:40.378966 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:41.116979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:41.116946 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 05:17:41.117756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:41.117732 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 05:17:41.378932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:41.378834 4425 patch_prober.go:29] interesting pod/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 05:17:41.378932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:41.378897 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 05:17:42.382883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:42.382848 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:17:42.383324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:42.383210 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:17:42.383919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:42.383899 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:17:52.454472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.454433 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l"] Nov 01 05:17:52.454967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.454953 4425 cpu_manager.go:414] "containerMap: removing container" podUID="67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" containerName="revoke-keys" Nov 01 05:17:52.455011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.454967 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" containerName="revoke-keys" Nov 01 05:17:52.455056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.455046 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" containerName="revoke-keys" Nov 01 05:17:52.458431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.458410 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.511989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.511929 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.512199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.512003 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.512199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.512154 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfj4\" (UniqueName: \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.512752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.512713 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:17:52.563343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.563304 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:17:52.563343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.563343 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:17:52.563343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.563315 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:17:52.566594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.566562 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l"] Nov 01 05:17:52.613548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.613515 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfj4\" (UniqueName: \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.613763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.613675 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.613763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.613724 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.614488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.614463 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.616326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.616303 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.675833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.675797 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfj4\" (UniqueName: \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\") pod \"maas-api-revoke-sub-a50e6159-sdp8l\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:52.771729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:52.771652 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:53.149235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:53.149205 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l"] Nov 01 05:17:53.150385 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:17:53.150360 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d367029_df40_4140_a20c_1aec0a3db4bd.slice/crio-0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6 WatchSource:0}: Error finding container 0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6: Status 404 returned error can't find the container with id 0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6 Nov 01 05:17:53.282624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:53.282594 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" event={"ID":"1d367029-df40-4140-a20c-1aec0a3db4bd","Type":"ContainerStarted","Data":"0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6"} Nov 01 05:17:54.290096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:54.290056 4425 generic.go:356] "Generic (PLEG): container finished" podID="1d367029-df40-4140-a20c-1aec0a3db4bd" containerID="845d3632aaa38709ebc77d260ae241128d4441559a7ed0f0c2e8815c822ddefe" exitCode=0 Nov 01 05:17:54.290535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:54.290144 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" event={"ID":"1d367029-df40-4140-a20c-1aec0a3db4bd","Type":"ContainerDied","Data":"845d3632aaa38709ebc77d260ae241128d4441559a7ed0f0c2e8815c822ddefe"} Nov 01 05:17:55.435021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.434993 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:55.543521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.543484 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\") pod \"1d367029-df40-4140-a20c-1aec0a3db4bd\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " Nov 01 05:17:55.543717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.543655 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\") pod \"1d367029-df40-4140-a20c-1aec0a3db4bd\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " Nov 01 05:17:55.543717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.543678 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\" (UniqueName: \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\") pod \"1d367029-df40-4140-a20c-1aec0a3db4bd\" (UID: \"1d367029-df40-4140-a20c-1aec0a3db4bd\") " Nov 01 05:17:55.544043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.544019 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca" pod "1d367029-df40-4140-a20c-1aec0a3db4bd" (UID: "1d367029-df40-4140-a20c-1aec0a3db4bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:17:55.545843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.545814 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca" pod "1d367029-df40-4140-a20c-1aec0a3db4bd" (UID: "1d367029-df40-4140-a20c-1aec0a3db4bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:17:55.546022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.546004 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4" pod "1d367029-df40-4140-a20c-1aec0a3db4bd" (UID: "1d367029-df40-4140-a20c-1aec0a3db4bd"). InnerVolumeSpecName "kube-api-access-kjfj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:17:55.644841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.644791 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:55.644841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.644831 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfj4\" (UniqueName: \"kubernetes.io/projected/1d367029-df40-4140-a20c-1aec0a3db4bd-kube-api-access-kjfj4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:55.644841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:55.644846 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d367029-df40-4140-a20c-1aec0a3db4bd-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:17:56.303108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:56.303068 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" event={"ID":"1d367029-df40-4140-a20c-1aec0a3db4bd","Type":"ContainerDied","Data":"0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6"} Nov 01 05:17:56.303313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:56.303123 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0810cc61ece17ccc435b21e8b09a37bb757bd84b703000208d7309e6f4bc1db6" Nov 01 05:17:56.303313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:17:56.303177 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l" Nov 01 05:17:56.948074 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:17:56.948033 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:18:08.652450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.652420 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg"] Nov 01 05:18:08.652953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.652912 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1d367029-df40-4140-a20c-1aec0a3db4bd" containerName="revoke-keys" Nov 01 05:18:08.652953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.652924 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d367029-df40-4140-a20c-1aec0a3db4bd" containerName="revoke-keys" Nov 01 05:18:08.653053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.653041 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d367029-df40-4140-a20c-1aec0a3db4bd" containerName="revoke-keys" Nov 01 05:18:08.657520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.657498 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.714037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.714005 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg"] Nov 01 05:18:08.765193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.765154 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:18:08.765193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.765177 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:18:08.765491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.765296 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:18:08.765491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.765336 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:18:08.782558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.782524 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtml\" (UniqueName: \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.782714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.782598 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.782759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.782729 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.883883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.883848 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.884053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.883890 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.884053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.883979 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtml\" (UniqueName: \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.884726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.884701 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.886573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.886549 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.946196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.946123 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtml\" (UniqueName: \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\") pod \"maas-api-revoke-sub-cef4940a-pbdxg\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:08.971587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:08.971561 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:09.344671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:09.344643 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg"] Nov 01 05:18:09.345798 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:09.345772 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2b998c_e641_4a9d_96fe_0d90911dc70e.slice/crio-0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a WatchSource:0}: Error finding container 0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a: Status 404 returned error can't find the container with id 0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a Nov 01 05:18:09.376683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:09.376652 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" event={"ID":"9b2b998c-e641-4a9d-96fe-0d90911dc70e","Type":"ContainerStarted","Data":"0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a"} Nov 01 05:18:10.376969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.376937 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59"] Nov 01 05:18:10.380874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.380854 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.383737 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.383708 4425 generic.go:356] "Generic (PLEG): container finished" podID="9b2b998c-e641-4a9d-96fe-0d90911dc70e" containerID="14f51f0ab2ec66d588cbfc0616adfc0155d7cd557b2e0a7b9dbaa7a171bd455b" exitCode=0 Nov 01 05:18:10.383839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.383783 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" event={"ID":"9b2b998c-e641-4a9d-96fe-0d90911dc70e","Type":"ContainerDied","Data":"14f51f0ab2ec66d588cbfc0616adfc0155d7cd557b2e0a7b9dbaa7a171bd455b"} Nov 01 05:18:10.438742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.438713 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59"] Nov 01 05:18:10.500153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.500054 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.500355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.500160 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxsz\" (UniqueName: \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.500355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.500322 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.601730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.601694 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.601928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.601776 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.601928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.601822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxsz\" (UniqueName: \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.602585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.602564 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.604319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.604298 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.663533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.663504 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxsz\" (UniqueName: \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\") pod \"maas-api-revoke-sub-237486cd-c4g59\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:10.694402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:10.694374 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:11.079429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.079401 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59"] Nov 01 05:18:11.079829 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:11.079804 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694fe92c_a8f4_4f35_93bd_9ec7a73061ef.slice/crio-f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7 WatchSource:0}: Error finding container f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7: Status 404 returned error can't find the container with id f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7 Nov 01 05:18:11.390960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.390930 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" event={"ID":"694fe92c-a8f4-4f35-93bd-9ec7a73061ef","Type":"ContainerStarted","Data":"98f6e315a427c514afe310391217337abf836e8597c76e14f4a3d56a6ea97985"} Nov 01 05:18:11.391372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.390983 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" event={"ID":"694fe92c-a8f4-4f35-93bd-9ec7a73061ef","Type":"ContainerStarted","Data":"f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7"} Nov 01 05:18:11.512369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.512318 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" podStartSLOduration=1.51230398 podStartE2EDuration="1.51230398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:11.509462999 +0000 UTC m=+906.650051953" watchObservedRunningTime="2026-11-01 05:18:11.51230398 +0000 UTC m=+906.652892896" Nov 01 05:18:11.538901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.538875 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:11.714039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.713947 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\" (UniqueName: \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\") pod \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " Nov 01 05:18:11.714201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.714041 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\") pod \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " Nov 01 05:18:11.714201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.714173 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\") pod \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\" (UID: \"9b2b998c-e641-4a9d-96fe-0d90911dc70e\") " Nov 01 05:18:11.714552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.714529 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca" pod "9b2b998c-e641-4a9d-96fe-0d90911dc70e" (UID: "9b2b998c-e641-4a9d-96fe-0d90911dc70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:18:11.716284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.716238 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml" pod "9b2b998c-e641-4a9d-96fe-0d90911dc70e" (UID: "9b2b998c-e641-4a9d-96fe-0d90911dc70e"). InnerVolumeSpecName "kube-api-access-nhtml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:18:11.716372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.716287 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca" pod "9b2b998c-e641-4a9d-96fe-0d90911dc70e" (UID: "9b2b998c-e641-4a9d-96fe-0d90911dc70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:18:11.815152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.815115 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:11.815152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.815146 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtml\" (UniqueName: \"kubernetes.io/projected/9b2b998c-e641-4a9d-96fe-0d90911dc70e-kube-api-access-nhtml\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:11.815152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:11.815156 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2b998c-e641-4a9d-96fe-0d90911dc70e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:12.398410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:12.398372 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" event={"ID":"9b2b998c-e641-4a9d-96fe-0d90911dc70e","Type":"ContainerDied","Data":"0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a"} Nov 01 05:18:12.398852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:12.398432 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee2665faefe00d718b893c09a2d89836d775f0feac4e62188ede3fb0959ce8a" Nov 01 05:18:12.398852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:12.398383 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg" Nov 01 05:18:12.400302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:12.400267 4425 generic.go:356] "Generic (PLEG): container finished" podID="694fe92c-a8f4-4f35-93bd-9ec7a73061ef" containerID="98f6e315a427c514afe310391217337abf836e8597c76e14f4a3d56a6ea97985" exitCode=0 Nov 01 05:18:12.400427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:12.400349 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" event={"ID":"694fe92c-a8f4-4f35-93bd-9ec7a73061ef","Type":"ContainerDied","Data":"98f6e315a427c514afe310391217337abf836e8597c76e14f4a3d56a6ea97985"} Nov 01 05:18:13.555670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.555641 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:13.642316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.642284 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\" (UniqueName: \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\") pod \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " Nov 01 05:18:13.642501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.642327 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\") pod \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " Nov 01 05:18:13.642501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.642438 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\") pod \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\" (UID: \"694fe92c-a8f4-4f35-93bd-9ec7a73061ef\") " Nov 01 05:18:13.642782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.642751 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca" pod "694fe92c-a8f4-4f35-93bd-9ec7a73061ef" (UID: "694fe92c-a8f4-4f35-93bd-9ec7a73061ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:18:13.642935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.642868 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:13.644615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.644585 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca" pod "694fe92c-a8f4-4f35-93bd-9ec7a73061ef" (UID: "694fe92c-a8f4-4f35-93bd-9ec7a73061ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:18:13.644701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.644666 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz" pod "694fe92c-a8f4-4f35-93bd-9ec7a73061ef" (UID: "694fe92c-a8f4-4f35-93bd-9ec7a73061ef"). InnerVolumeSpecName "kube-api-access-qjxsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:18:13.743577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.743501 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxsz\" (UniqueName: \"kubernetes.io/projected/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-kube-api-access-qjxsz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:13.743577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:13.743529 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/694fe92c-a8f4-4f35-93bd-9ec7a73061ef-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:14.413267 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:14.413228 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" event={"ID":"694fe92c-a8f4-4f35-93bd-9ec7a73061ef","Type":"ContainerDied","Data":"f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7"} Nov 01 05:18:14.413438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:14.413302 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f089e9dccf6946e5162f52753950b9a7d08598ab0c381a1f5dc630809ff732c7" Nov 01 05:18:14.413438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:14.413239 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59" Nov 01 05:18:26.948292 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:18:26.948233 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:18:29.088430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.088390 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4"] Nov 01 05:18:29.088909 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.088895 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9b2b998c-e641-4a9d-96fe-0d90911dc70e" containerName="revoke-keys" Nov 01 05:18:29.088952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.088910 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2b998c-e641-4a9d-96fe-0d90911dc70e" containerName="revoke-keys" Nov 01 05:18:29.088952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.088931 4425 cpu_manager.go:414] "containerMap: removing container" podUID="694fe92c-a8f4-4f35-93bd-9ec7a73061ef" containerName="revoke-keys" Nov 01 05:18:29.088952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.088936 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694fe92c-a8f4-4f35-93bd-9ec7a73061ef" containerName="revoke-keys" Nov 01 05:18:29.089050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.089001 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2b998c-e641-4a9d-96fe-0d90911dc70e" containerName="revoke-keys" Nov 01 05:18:29.089050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.089011 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="694fe92c-a8f4-4f35-93bd-9ec7a73061ef" containerName="revoke-keys" Nov 01 05:18:29.092339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.092324 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.197451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.197414 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:18:29.197451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.197448 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:18:29.197634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.197445 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:18:29.197634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.197487 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:18:29.200067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.200046 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4"] Nov 01 05:18:29.206486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.206466 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hm2n\" (UniqueName: \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.206549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.206535 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.206634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.206617 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.307631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.307592 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hm2n\" (UniqueName: \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.307829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.307687 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.307829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.307720 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.308365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.308342 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.310512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.310489 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.369719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.369653 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hm2n\" (UniqueName: \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\") pod \"maas-api-revoke-sub-8ce61f67-jp8w4\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.406272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.406230 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:29.780813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:29.780787 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4"] Nov 01 05:18:29.782284 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:29.782237 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98a7a0c_f4a2_49c0_be2d_f7235cd833bb.slice/crio-38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41 WatchSource:0}: Error finding container 38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41: Status 404 returned error can't find the container with id 38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41 Nov 01 05:18:30.505320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:30.505286 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" event={"ID":"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb","Type":"ContainerStarted","Data":"64f27810e3d64c325370757f4e098f28a604da6481406ac7833215c4e9ef9704"} Nov 01 05:18:30.505795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:30.505345 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" event={"ID":"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb","Type":"ContainerStarted","Data":"38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41"} Nov 01 05:18:30.624901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:30.624840 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" podStartSLOduration=1.624825043 podStartE2EDuration="1.624825043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:30.622293709 +0000 UTC m=+925.762882626" watchObservedRunningTime="2026-11-01 05:18:30.624825043 +0000 UTC m=+925.765413962" Nov 01 05:18:31.513441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:31.513401 4425 generic.go:356] "Generic (PLEG): container finished" podID="a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" containerID="64f27810e3d64c325370757f4e098f28a604da6481406ac7833215c4e9ef9704" exitCode=0 Nov 01 05:18:31.513900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:31.513480 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" event={"ID":"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb","Type":"ContainerDied","Data":"64f27810e3d64c325370757f4e098f28a604da6481406ac7833215c4e9ef9704"} Nov 01 05:18:32.657992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.657967 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:32.748245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.748214 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\") pod \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " Nov 01 05:18:32.748437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.748362 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\") pod \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " Nov 01 05:18:32.748437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.748405 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\" (UniqueName: \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\") pod \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\" (UID: \"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb\") " Nov 01 05:18:32.748632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.748603 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca" pod "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" (UID: "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:18:32.748804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.748785 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:32.750774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.750739 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n" pod "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" (UID: "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb"). InnerVolumeSpecName "kube-api-access-6hm2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:18:32.750774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.750747 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca" pod "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" (UID: "a98a7a0c-f4a2-49c0-be2d-f7235cd833bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:18:32.849938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.849903 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:32.849938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:32.849935 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hm2n\" (UniqueName: \"kubernetes.io/projected/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb-kube-api-access-6hm2n\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:33.524924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:33.524890 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" event={"ID":"a98a7a0c-f4a2-49c0-be2d-f7235cd833bb","Type":"ContainerDied","Data":"38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41"} Nov 01 05:18:33.525103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:33.524929 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4" Nov 01 05:18:33.525103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:33.524947 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38e98629d4cc94422beedb7fb2fedc2c425669ea581b9e93e47818a23a06bd41" Nov 01 05:18:36.869609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.869576 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:18:36.870216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.870196 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" containerName="revoke-keys" Nov 01 05:18:36.870279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.870219 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" containerName="revoke-keys" Nov 01 05:18:36.870331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.870320 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" containerName="revoke-keys" Nov 01 05:18:36.873990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.873972 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:36.927575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.927549 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\"" Nov 01 05:18:36.981076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.981048 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:18:36.999117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999014 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999393 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999474 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999568 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999727 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmxk\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999795 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999835 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999879 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.003273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:36.999915 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101188 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101279 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101333 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101351 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101409 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmxk\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101448 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101475 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101517 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101543 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101852 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.101952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101902 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.102010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.101976 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.102167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.102003 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.102316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.102241 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.103928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.103904 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.104408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.104390 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.164054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.163980 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.164185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.164091 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmxk\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.187888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.187862 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:37.618432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.618404 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:18:37.620849 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:37.620823 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593c1831_d654_412a_b1bf_e125ed51154c.slice/crio-52627ac1dfb4e41c41e0f40c28253304b7de5617fb34be465add2830c2361af8 WatchSource:0}: Error finding container 52627ac1dfb4e41c41e0f40c28253304b7de5617fb34be465add2830c2361af8: Status 404 returned error can't find the container with id 52627ac1dfb4e41c41e0f40c28253304b7de5617fb34be465add2830c2361af8 Nov 01 05:18:37.623275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.623223 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:37.623358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.623314 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:37.623358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:37.623344 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:38.159912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.159878 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 05:18:38.163792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.163775 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.218841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.218807 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"openshift-service-ca.crt\"" Nov 01 05:18:38.221995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.221966 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 05:18:38.269628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.269570 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"kube-root-ca.crt\"" Nov 01 05:18:38.269628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.269620 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"distinct-w4-a48099-kserve-self-signed-certs\"" Nov 01 05:18:38.269832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.269777 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"default-dockercfg-r7p77\"" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.312982 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.313073 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.313189 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkbr\" (UniqueName: \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.313223 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.313284 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.314150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.313310 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.414912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.414803 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.414912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.414885 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415104 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkbr\" (UniqueName: \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415142 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415219 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415245 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415487 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.415678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.415658 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.416084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.416060 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.418407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.418369 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.418794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.418775 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.477687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.477648 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkbr\" (UniqueName: \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\") pod \"distinct-w4-a48099-kserve-55547bd958-dqcg8\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:38.586097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.586067 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" event={"ID":"593c1831-d654-412a-b1bf-e125ed51154c","Type":"ContainerStarted","Data":"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8"} Nov 01 05:18:38.586302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.586122 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" event={"ID":"593c1831-d654-412a-b1bf-e125ed51154c","Type":"ContainerStarted","Data":"52627ac1dfb4e41c41e0f40c28253304b7de5617fb34be465add2830c2361af8"} Nov 01 05:18:38.709154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.709038 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podStartSLOduration=2.709022118 podStartE2EDuration="2.709022118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:38.707660774 +0000 UTC m=+933.848249696" watchObservedRunningTime="2026-11-01 05:18:38.709022118 +0000 UTC m=+933.849611037" Nov 01 05:18:38.775869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:38.775835 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:39.166222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.166180 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 05:18:39.170647 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:39.170608 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40481906_1eb4_484a_81f7_65ab5940aaca.slice/crio-11c0689ceec61d94209ed69a4178f7dc0d2dc1fdea5acca735bb3c6cc1043f84 WatchSource:0}: Error finding container 11c0689ceec61d94209ed69a4178f7dc0d2dc1fdea5acca735bb3c6cc1043f84: Status 404 returned error can't find the container with id 11c0689ceec61d94209ed69a4178f7dc0d2dc1fdea5acca735bb3c6cc1043f84 Nov 01 05:18:39.188378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.188346 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:39.190466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.190441 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:39.190582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.190490 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:39.594155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.593882 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" event={"ID":"40481906-1eb4-484a-81f7-65ab5940aaca","Type":"ContainerStarted","Data":"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee"} Nov 01 05:18:39.594155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.593942 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" event={"ID":"40481906-1eb4-484a-81f7-65ab5940aaca","Type":"ContainerStarted","Data":"11c0689ceec61d94209ed69a4178f7dc0d2dc1fdea5acca735bb3c6cc1043f84"} Nov 01 05:18:39.732929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.732876 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podStartSLOduration=1.73286326 podStartE2EDuration="1.73286326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:39.731572317 +0000 UTC m=+934.872161255" watchObservedRunningTime="2026-11-01 05:18:39.73286326 +0000 UTC m=+934.873452179" Nov 01 05:18:39.776317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:39.776282 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:40.188691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:40.188651 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:40.189282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:40.188716 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:41.188445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.188408 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:41.188641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.188458 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:41.220308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.220273 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq"] Nov 01 05:18:41.224372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.224356 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.282365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.282297 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:18:41.298370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.298343 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:18:41.332736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.332707 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:18:41.332897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.332787 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:18:41.335844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.335823 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq"] Nov 01 05:18:41.371004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.370964 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.371199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.371068 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.371199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.371163 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szsbk\" (UniqueName: \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.472237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.472136 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsbk\" (UniqueName: \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.472456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.472343 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.472456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.472400 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.472963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.472935 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.474916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.474899 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.534655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.534614 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsbk\" (UniqueName: \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\") pod \"maas-api-revoke-sub-937c8bea-gthjq\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.537403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.537383 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:41.968615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:41.968483 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq"] Nov 01 05:18:41.973392 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:41.973352 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d60ee8b_77c2_4230_8bd9_68148ec52722.slice/crio-37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76 WatchSource:0}: Error finding container 37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76: Status 404 returned error can't find the container with id 37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76 Nov 01 05:18:42.188535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:42.188510 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:42.188661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:42.188557 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:42.614937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:42.614902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" event={"ID":"7d60ee8b-77c2-4230-8bd9-68148ec52722","Type":"ContainerStarted","Data":"7e11b82e63e23172a64ce4d076a7c4c911b1ea2a09b6393ee8271322e2cc7ea7"} Nov 01 05:18:42.615415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:42.614960 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" event={"ID":"7d60ee8b-77c2-4230-8bd9-68148ec52722","Type":"ContainerStarted","Data":"37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76"} Nov 01 05:18:42.771915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:42.771864 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" podStartSLOduration=1.771850296 podStartE2EDuration="1.771850296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:42.769678775 +0000 UTC m=+937.910267697" watchObservedRunningTime="2026-11-01 05:18:42.771850296 +0000 UTC m=+937.912439215" Nov 01 05:18:43.188759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:43.188722 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:43.188931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:43.188780 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:43.622010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:43.621979 4425 generic.go:356] "Generic (PLEG): container finished" podID="7d60ee8b-77c2-4230-8bd9-68148ec52722" containerID="7e11b82e63e23172a64ce4d076a7c4c911b1ea2a09b6393ee8271322e2cc7ea7" exitCode=0 Nov 01 05:18:43.622468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:43.622058 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" event={"ID":"7d60ee8b-77c2-4230-8bd9-68148ec52722","Type":"ContainerDied","Data":"7e11b82e63e23172a64ce4d076a7c4c911b1ea2a09b6393ee8271322e2cc7ea7"} Nov 01 05:18:44.189196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.189153 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 05:18:44.189392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.189234 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 05:18:44.785717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.785691 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:44.911687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.911612 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\") pod \"7d60ee8b-77c2-4230-8bd9-68148ec52722\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " Nov 01 05:18:44.911854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.911696 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\" (UniqueName: \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\") pod \"7d60ee8b-77c2-4230-8bd9-68148ec52722\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " Nov 01 05:18:44.911854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.911738 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\") pod \"7d60ee8b-77c2-4230-8bd9-68148ec52722\" (UID: \"7d60ee8b-77c2-4230-8bd9-68148ec52722\") " Nov 01 05:18:44.912375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.912332 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca" pod "7d60ee8b-77c2-4230-8bd9-68148ec52722" (UID: "7d60ee8b-77c2-4230-8bd9-68148ec52722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:18:44.914105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.914078 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk" pod "7d60ee8b-77c2-4230-8bd9-68148ec52722" (UID: "7d60ee8b-77c2-4230-8bd9-68148ec52722"). InnerVolumeSpecName "kube-api-access-szsbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:18:44.914430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:44.914407 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca" pod "7d60ee8b-77c2-4230-8bd9-68148ec52722" (UID: "7d60ee8b-77c2-4230-8bd9-68148ec52722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:18:45.013495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.013453 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:45.013495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.013490 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szsbk\" (UniqueName: \"kubernetes.io/projected/7d60ee8b-77c2-4230-8bd9-68148ec52722-kube-api-access-szsbk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:45.013675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.013504 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d60ee8b-77c2-4230-8bd9-68148ec52722-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:45.193020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.192936 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:45.193451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.193430 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:45.194173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.194154 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:18:45.433953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.433913 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p"] Nov 01 05:18:45.434530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.434510 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7d60ee8b-77c2-4230-8bd9-68148ec52722" containerName="revoke-keys" Nov 01 05:18:45.434652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.434531 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d60ee8b-77c2-4230-8bd9-68148ec52722" containerName="revoke-keys" Nov 01 05:18:45.434742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.434728 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d60ee8b-77c2-4230-8bd9-68148ec52722" containerName="revoke-keys" Nov 01 05:18:45.438281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.438234 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.495786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.495742 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p"] Nov 01 05:18:45.519122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.519087 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.519346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.519153 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.519346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.519244 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhx95\" (UniqueName: \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.621083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.621043 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.621316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.621096 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.621316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.621162 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhx95\" (UniqueName: \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.621781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.621759 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.623708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.623685 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.636609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.636589 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" Nov 01 05:18:45.636717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.636605 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq" event={"ID":"7d60ee8b-77c2-4230-8bd9-68148ec52722","Type":"ContainerDied","Data":"37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76"} Nov 01 05:18:45.636717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.636655 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a0f332ca0300767b4f37dc19c6f4fbcde44302da99cf23e0e0120f3f326a76" Nov 01 05:18:45.683828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.683799 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhx95\" (UniqueName: \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\") pod \"maas-api-revoke-sub-78055817-bzn5p\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:45.753578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:45.753503 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:46.131461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:46.131432 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p"] Nov 01 05:18:46.132013 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:46.131991 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dab1e00_deca_475c_ba24_2e0177671a59.slice/crio-184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639 WatchSource:0}: Error finding container 184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639: Status 404 returned error can't find the container with id 184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639 Nov 01 05:18:46.646613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:46.646563 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" event={"ID":"6dab1e00-deca-475c-ba24-2e0177671a59","Type":"ContainerStarted","Data":"853e9454849c98a47704721afc4be8aa67d6b96b7f0eba05adfe43e064434c24"} Nov 01 05:18:46.646805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:46.646652 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" event={"ID":"6dab1e00-deca-475c-ba24-2e0177671a59","Type":"ContainerStarted","Data":"184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639"} Nov 01 05:18:46.763486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:46.763363 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" podStartSLOduration=1.7633444150000002 podStartE2EDuration="1.763344415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:46.763026109 +0000 UTC m=+941.903615028" watchObservedRunningTime="2026-11-01 05:18:46.763344415 +0000 UTC m=+941.903933339" Nov 01 05:18:47.128008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.127970 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 05:18:47.131929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.131912 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.186093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.186056 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2fd4c9\"" Nov 01 05:18:47.186549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.186520 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2fd4c9-dockercfg-pxh9b\"" Nov 01 05:18:47.189989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.189967 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 05:18:47.246515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.244128 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtn7\" (UniqueName: \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.246515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.244208 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.345343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.345289 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.345568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.345547 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtn7\" (UniqueName: \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.346323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.346299 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.408199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.408162 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtn7\" (UniqueName: \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\") pod \"payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.445169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.445125 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:47.666414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.666319 4425 generic.go:356] "Generic (PLEG): container finished" podID="6dab1e00-deca-475c-ba24-2e0177671a59" containerID="853e9454849c98a47704721afc4be8aa67d6b96b7f0eba05adfe43e064434c24" exitCode=0 Nov 01 05:18:47.666414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.666369 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" event={"ID":"6dab1e00-deca-475c-ba24-2e0177671a59","Type":"ContainerDied","Data":"853e9454849c98a47704721afc4be8aa67d6b96b7f0eba05adfe43e064434c24"} Nov 01 05:18:47.826765 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:47.826734 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa7b5c_1fac_456b_b04d_702ff8a050c4.slice/crio-8b44f4d3d441b986b7e1f7331d16ce75c7983447ca241e558578430e415a49d1 WatchSource:0}: Error finding container 8b44f4d3d441b986b7e1f7331d16ce75c7983447ca241e558578430e415a49d1: Status 404 returned error can't find the container with id 8b44f4d3d441b986b7e1f7331d16ce75c7983447ca241e558578430e415a49d1 Nov 01 05:18:47.827413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:47.827389 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 05:18:48.674926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.674888 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" event={"ID":"5cfa7b5c-1fac-456b-b04d-702ff8a050c4","Type":"ContainerStarted","Data":"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3"} Nov 01 05:18:48.675386 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.674948 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" event={"ID":"5cfa7b5c-1fac-456b-b04d-702ff8a050c4","Type":"ContainerStarted","Data":"8b44f4d3d441b986b7e1f7331d16ce75c7983447ca241e558578430e415a49d1"} Nov 01 05:18:48.675386 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.675115 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:18:48.785764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.781622 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:48.814157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.814058 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" podStartSLOduration=1.8140363769999999 podStartE2EDuration="1.814036377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:48.806194756 +0000 UTC m=+943.946783676" watchObservedRunningTime="2026-11-01 05:18:48.814036377 +0000 UTC m=+943.954625301" Nov 01 05:18:48.821336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.820141 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:48.888438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.888404 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:48.987833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.987747 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\") pod \"6dab1e00-deca-475c-ba24-2e0177671a59\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " Nov 01 05:18:48.987993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.987836 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\") pod \"6dab1e00-deca-475c-ba24-2e0177671a59\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " Nov 01 05:18:48.987993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.987888 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\" (UniqueName: \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\") pod \"6dab1e00-deca-475c-ba24-2e0177671a59\" (UID: \"6dab1e00-deca-475c-ba24-2e0177671a59\") " Nov 01 05:18:48.988823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.988781 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca" pod "6dab1e00-deca-475c-ba24-2e0177671a59" (UID: "6dab1e00-deca-475c-ba24-2e0177671a59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:18:48.996213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.995747 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca" pod "6dab1e00-deca-475c-ba24-2e0177671a59" (UID: "6dab1e00-deca-475c-ba24-2e0177671a59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:18:48.996213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:48.995780 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95" pod "6dab1e00-deca-475c-ba24-2e0177671a59" (UID: "6dab1e00-deca-475c-ba24-2e0177671a59"). InnerVolumeSpecName "kube-api-access-fhx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:18:49.089112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.089072 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:49.089408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.089370 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dab1e00-deca-475c-ba24-2e0177671a59-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:49.089408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.089406 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhx95\" (UniqueName: \"kubernetes.io/projected/6dab1e00-deca-475c-ba24-2e0177671a59-kube-api-access-fhx95\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:18:49.689276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.688990 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" event={"ID":"6dab1e00-deca-475c-ba24-2e0177671a59","Type":"ContainerDied","Data":"184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639"} Nov 01 05:18:49.689276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.689056 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184c684ff1e94ecd53a9a0d4399118420cc7ad8b0e993255a3d8eaabc97c6639" Nov 01 05:18:49.689834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.689347 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p" Nov 01 05:18:49.704245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:49.703997 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 05:18:53.842311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.842243 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:18:53.843025 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.843009 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6dab1e00-deca-475c-ba24-2e0177671a59" containerName="revoke-keys" Nov 01 05:18:53.843078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.843028 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dab1e00-deca-475c-ba24-2e0177671a59" containerName="revoke-keys" Nov 01 05:18:53.843194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.843183 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dab1e00-deca-475c-ba24-2e0177671a59" containerName="revoke-keys" Nov 01 05:18:53.846927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.846909 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.900454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.900422 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3d70ff-openshift-default-dockercfg-x8xrx\"" Nov 01 05:18:53.954070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954031 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:18:53.954240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954130 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vccj\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954168 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954191 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954337 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954397 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954495 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954563 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954634 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:53.954688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:53.954662 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.055704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.055656 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.055704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.055705 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.055972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.055836 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vccj\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.055972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.055891 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.055972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.055946 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056009 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056091 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056304 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056430 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056455 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056494 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056521 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056559 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.056992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.056965 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.058468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.058450 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.058676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.058654 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.117856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.117791 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.117972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.117927 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vccj\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.159833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.159806 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:54.588283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.588238 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:18:54.589295 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:54.589268 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e9bf74_070f_4c59_ba1f_22d2209c488e.slice/crio-d945134f79f1d1b9492bb8c5976db064fd6f726e6509aad9c21f46aeb1cab4ce WatchSource:0}: Error finding container d945134f79f1d1b9492bb8c5976db064fd6f726e6509aad9c21f46aeb1cab4ce: Status 404 returned error can't find the container with id d945134f79f1d1b9492bb8c5976db064fd6f726e6509aad9c21f46aeb1cab4ce Nov 01 05:18:54.591695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.591659 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:54.591745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.591728 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:54.591781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.591759 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:18:54.722534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.722500 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" event={"ID":"17e9bf74-070f-4c59-ba1f-22d2209c488e","Type":"ContainerStarted","Data":"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32"} Nov 01 05:18:54.722729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.722560 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" event={"ID":"17e9bf74-070f-4c59-ba1f-22d2209c488e","Type":"ContainerStarted","Data":"d945134f79f1d1b9492bb8c5976db064fd6f726e6509aad9c21f46aeb1cab4ce"} Nov 01 05:18:54.846518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:54.846412 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podStartSLOduration=1.846398685 podStartE2EDuration="1.846398685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:54.844394285 +0000 UTC m=+949.984983216" watchObservedRunningTime="2026-11-01 05:18:54.846398685 +0000 UTC m=+949.986987604" Nov 01 05:18:55.164089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:55.162645 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:18:55.165608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:55.165577 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:18:55.165810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:55.165782 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:18:56.166283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.165696 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:18:56.166283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.165760 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:18:56.864278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.863447 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn"] Nov 01 05:18:56.868563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.868520 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.888918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.888892 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn6t\" (UniqueName: \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.889032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.888939 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.889072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.889053 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.904541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.904511 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn"] Nov 01 05:18:56.966388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.955590 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:18:56.966388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.955750 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:18:56.966388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.956549 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:18:56.967439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.967407 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:18:56.976853 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:18:56.976817 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:18:56.990400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.990367 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.990535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.990445 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.990658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.990639 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn6t\" (UniqueName: \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.991212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.991192 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:56.992980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:56.992962 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:57.067815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.067771 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn6t\" (UniqueName: \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\") pod \"maas-api-revoke-sub-3928d7fd-8c4hn\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:57.161338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.160681 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:18:57.161338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.160744 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:18:57.182468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.182433 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:18:57.578363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.578328 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn"] Nov 01 05:18:57.579182 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:18:57.579154 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1a55c1_f296_497b_a8e0_46bd8450f606.slice/crio-8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5 WatchSource:0}: Error finding container 8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5: Status 404 returned error can't find the container with id 8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5 Nov 01 05:18:57.773424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:57.773388 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" event={"ID":"ec1a55c1-f296-497b-a8e0-46bd8450f606","Type":"ContainerStarted","Data":"8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5"} Nov 01 05:18:58.160798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:58.160764 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:18:58.161072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:58.161031 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:18:58.782234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:58.782196 4425 generic.go:356] "Generic (PLEG): container finished" podID="ec1a55c1-f296-497b-a8e0-46bd8450f606" containerID="a002517631f52231cbec1958e6a8009211237f86df3c75a5301544f4f13b5c74" exitCode=0 Nov 01 05:18:58.782658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:58.782297 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" event={"ID":"ec1a55c1-f296-497b-a8e0-46bd8450f606","Type":"ContainerDied","Data":"a002517631f52231cbec1958e6a8009211237f86df3c75a5301544f4f13b5c74"} Nov 01 05:18:59.160722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:59.160682 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:18:59.161018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:59.160985 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:18:59.693394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:18:59.693365 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 05:19:00.004566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.004540 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:19:00.030079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.030046 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\" (UniqueName: \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\") pod \"ec1a55c1-f296-497b-a8e0-46bd8450f606\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " Nov 01 05:19:00.030079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.030084 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\") pod \"ec1a55c1-f296-497b-a8e0-46bd8450f606\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " Nov 01 05:19:00.030440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.030104 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\") pod \"ec1a55c1-f296-497b-a8e0-46bd8450f606\" (UID: \"ec1a55c1-f296-497b-a8e0-46bd8450f606\") " Nov 01 05:19:00.030524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.030495 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca" pod "ec1a55c1-f296-497b-a8e0-46bd8450f606" (UID: "ec1a55c1-f296-497b-a8e0-46bd8450f606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:00.032502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.032470 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca" pod "ec1a55c1-f296-497b-a8e0-46bd8450f606" (UID: "ec1a55c1-f296-497b-a8e0-46bd8450f606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:00.032502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.032486 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t" pod "ec1a55c1-f296-497b-a8e0-46bd8450f606" (UID: "ec1a55c1-f296-497b-a8e0-46bd8450f606"). InnerVolumeSpecName "kube-api-access-swn6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:00.131938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.131896 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn6t\" (UniqueName: \"kubernetes.io/projected/ec1a55c1-f296-497b-a8e0-46bd8450f606-kube-api-access-swn6t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:00.131938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.131939 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:00.132146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.131957 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1a55c1-f296-497b-a8e0-46bd8450f606-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:00.166766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.166724 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:19:00.167075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.167039 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:19:00.796459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.796427 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" Nov 01 05:19:00.796459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.796448 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn" event={"ID":"ec1a55c1-f296-497b-a8e0-46bd8450f606","Type":"ContainerDied","Data":"8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5"} Nov 01 05:19:00.796638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:00.796524 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bab58f71226c09c4eac60049a01195f0c9a3b1657d265ff474f64ece7d252e5" Nov 01 05:19:01.160671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.160639 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 05:19:01.161068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.160693 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 05:19:01.448621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.448517 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr"] Nov 01 05:19:01.449118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.449101 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ec1a55c1-f296-497b-a8e0-46bd8450f606" containerName="revoke-keys" Nov 01 05:19:01.449118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.449117 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1a55c1-f296-497b-a8e0-46bd8450f606" containerName="revoke-keys" Nov 01 05:19:01.449323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.449222 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1a55c1-f296-497b-a8e0-46bd8450f606" containerName="revoke-keys" Nov 01 05:19:01.452709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.452684 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.515068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.515018 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:01.515428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.515022 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:01.515677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.515656 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr"] Nov 01 05:19:01.549446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.549409 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkk5\" (UniqueName: \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.549446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.549448 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.549643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.549598 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.557354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.557327 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:01.557523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.557400 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:01.650689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.650645 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkk5\" (UniqueName: \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.650689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.650691 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.650927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.650771 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.651511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.651448 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.653330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.653306 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.713690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.713615 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkk5\" (UniqueName: \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\") pod \"maas-api-revoke-sub-36809939-5kgqr\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:01.767243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:01.767202 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:02.160106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.160070 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr"] Nov 01 05:19:02.160388 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:02.160353 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fecc27e_8d20_46b4_8e1d_218f3946c486.slice/crio-d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536 WatchSource:0}: Error finding container d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536: Status 404 returned error can't find the container with id d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536 Nov 01 05:19:02.165948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.165927 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:19:02.166512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.166482 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:19:02.167222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.167198 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:19:02.810982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.810952 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" event={"ID":"5fecc27e-8d20-46b4-8e1d-218f3946c486","Type":"ContainerStarted","Data":"ec811db705aae3a17391e9496040bf09565abc6a862eeb739b151ada410a6e90"} Nov 01 05:19:02.811119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.811013 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" event={"ID":"5fecc27e-8d20-46b4-8e1d-218f3946c486","Type":"ContainerStarted","Data":"d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536"} Nov 01 05:19:02.929869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:02.929818 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" podStartSLOduration=1.92980455 podStartE2EDuration="1.92980455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:02.928037541 +0000 UTC m=+958.068626481" watchObservedRunningTime="2026-11-01 05:19:02.92980455 +0000 UTC m=+958.070393469" Nov 01 05:19:03.818638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:03.818604 4425 generic.go:356] "Generic (PLEG): container finished" podID="5fecc27e-8d20-46b4-8e1d-218f3946c486" containerID="ec811db705aae3a17391e9496040bf09565abc6a862eeb739b151ada410a6e90" exitCode=0 Nov 01 05:19:03.819024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:03.818695 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" event={"ID":"5fecc27e-8d20-46b4-8e1d-218f3946c486","Type":"ContainerDied","Data":"ec811db705aae3a17391e9496040bf09565abc6a862eeb739b151ada410a6e90"} Nov 01 05:19:04.962372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:04.962343 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:05.097538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.097450 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\") pod \"5fecc27e-8d20-46b4-8e1d-218f3946c486\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " Nov 01 05:19:05.097708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.097690 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\" (UniqueName: \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\") pod \"5fecc27e-8d20-46b4-8e1d-218f3946c486\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " Nov 01 05:19:05.097759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.097728 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\") pod \"5fecc27e-8d20-46b4-8e1d-218f3946c486\" (UID: \"5fecc27e-8d20-46b4-8e1d-218f3946c486\") " Nov 01 05:19:05.098123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.098099 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca" pod "5fecc27e-8d20-46b4-8e1d-218f3946c486" (UID: "5fecc27e-8d20-46b4-8e1d-218f3946c486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:05.098204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.098179 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:05.099631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.099604 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca" pod "5fecc27e-8d20-46b4-8e1d-218f3946c486" (UID: "5fecc27e-8d20-46b4-8e1d-218f3946c486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:05.099956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.099939 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5" pod "5fecc27e-8d20-46b4-8e1d-218f3946c486" (UID: "5fecc27e-8d20-46b4-8e1d-218f3946c486"). InnerVolumeSpecName "kube-api-access-9hkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:05.199713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.199678 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkk5\" (UniqueName: \"kubernetes.io/projected/5fecc27e-8d20-46b4-8e1d-218f3946c486-kube-api-access-9hkk5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:05.199713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.199710 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fecc27e-8d20-46b4-8e1d-218f3946c486-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:05.836303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.836242 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" event={"ID":"5fecc27e-8d20-46b4-8e1d-218f3946c486","Type":"ContainerDied","Data":"d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536"} Nov 01 05:19:05.836303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.836285 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr" Nov 01 05:19:05.836573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:05.836320 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4411ebbe2a2829afbac080ab923cc6b3d55205ae524f59fafe0fbf0003af536" Nov 01 05:19:12.587843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.587811 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f"] Nov 01 05:19:12.588394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.588380 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5fecc27e-8d20-46b4-8e1d-218f3946c486" containerName="revoke-keys" Nov 01 05:19:12.588444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.588395 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fecc27e-8d20-46b4-8e1d-218f3946c486" containerName="revoke-keys" Nov 01 05:19:12.588553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.588543 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fecc27e-8d20-46b4-8e1d-218f3946c486" containerName="revoke-keys" Nov 01 05:19:12.593058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.593042 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.647082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.647047 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:12.649929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.649899 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f"] Nov 01 05:19:12.684062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.684016 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.684243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.684070 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqqh\" (UniqueName: \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.684243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.684212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.695658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.695628 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:12.695658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.695630 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:12.697443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.697424 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:12.787997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.787965 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.787997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.788001 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqqh\" (UniqueName: \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.788208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.788069 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.788855 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.788779 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.790494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.790472 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.850277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.850188 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqqh\" (UniqueName: \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\") pod \"maas-api-revoke-sub-a406644a-ng42f\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:12.907598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:12.907570 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:13.287376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:13.287345 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f"] Nov 01 05:19:13.288355 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:13.288330 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012ff719_bf23_4769_8331_e858420faf8d.slice/crio-a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276 WatchSource:0}: Error finding container a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276: Status 404 returned error can't find the container with id a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276 Nov 01 05:19:13.889083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:13.889052 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" event={"ID":"012ff719-bf23-4769-8331-e858420faf8d","Type":"ContainerStarted","Data":"dd66a0aea9cba83c1cda9867dd8d61cb3ac51995bae3ef7ce0bd06aa6b529e23"} Nov 01 05:19:13.889505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:13.889106 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" event={"ID":"012ff719-bf23-4769-8331-e858420faf8d","Type":"ContainerStarted","Data":"a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276"} Nov 01 05:19:14.008367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:14.008313 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" podStartSLOduration=2.008298523 podStartE2EDuration="2.008298523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:14.005675351 +0000 UTC m=+969.146264271" watchObservedRunningTime="2026-11-01 05:19:14.008298523 +0000 UTC m=+969.148887452" Nov 01 05:19:14.896494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:14.896460 4425 generic.go:356] "Generic (PLEG): container finished" podID="012ff719-bf23-4769-8331-e858420faf8d" containerID="dd66a0aea9cba83c1cda9867dd8d61cb3ac51995bae3ef7ce0bd06aa6b529e23" exitCode=0 Nov 01 05:19:14.896934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:14.896537 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" event={"ID":"012ff719-bf23-4769-8331-e858420faf8d","Type":"ContainerDied","Data":"dd66a0aea9cba83c1cda9867dd8d61cb3ac51995bae3ef7ce0bd06aa6b529e23"} Nov 01 05:19:16.054150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.054123 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:16.227008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.226921 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\") pod \"012ff719-bf23-4769-8331-e858420faf8d\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " Nov 01 05:19:16.227155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.227076 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\" (UniqueName: \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\") pod \"012ff719-bf23-4769-8331-e858420faf8d\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " Nov 01 05:19:16.227155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.227097 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\") pod \"012ff719-bf23-4769-8331-e858420faf8d\" (UID: \"012ff719-bf23-4769-8331-e858420faf8d\") " Nov 01 05:19:16.227627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.227596 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca" pod "012ff719-bf23-4769-8331-e858420faf8d" (UID: "012ff719-bf23-4769-8331-e858420faf8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:16.229199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.229169 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca" pod "012ff719-bf23-4769-8331-e858420faf8d" (UID: "012ff719-bf23-4769-8331-e858420faf8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:16.229318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.229245 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh" pod "012ff719-bf23-4769-8331-e858420faf8d" (UID: "012ff719-bf23-4769-8331-e858420faf8d"). InnerVolumeSpecName "kube-api-access-xpqqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:16.328082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.328040 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqqh\" (UniqueName: \"kubernetes.io/projected/012ff719-bf23-4769-8331-e858420faf8d-kube-api-access-xpqqh\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:16.328082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.328067 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ff719-bf23-4769-8331-e858420faf8d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:16.328082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.328077 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ff719-bf23-4769-8331-e858420faf8d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:16.910336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.910300 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" Nov 01 05:19:16.910541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.910306 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f" event={"ID":"012ff719-bf23-4769-8331-e858420faf8d","Type":"ContainerDied","Data":"a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276"} Nov 01 05:19:16.910541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:16.910425 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47db684946d5747e840804e9a3965f129aa72adce9a66e276186626d3baf276" Nov 01 05:19:18.521673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.521638 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv"] Nov 01 05:19:18.522226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.522212 4425 cpu_manager.go:414] "containerMap: removing container" podUID="012ff719-bf23-4769-8331-e858420faf8d" containerName="revoke-keys" Nov 01 05:19:18.522288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.522227 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012ff719-bf23-4769-8331-e858420faf8d" containerName="revoke-keys" Nov 01 05:19:18.522368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.522358 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="012ff719-bf23-4769-8331-e858420faf8d" containerName="revoke-keys" Nov 01 05:19:18.525961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.525946 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.582993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.582958 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv"] Nov 01 05:19:18.630146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.630103 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:18.630146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.630140 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:18.630767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.630735 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:18.630885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.630770 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:18.657094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.657059 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.657327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.657141 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdgg\" (UniqueName: \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.657327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.657235 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.758555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.758515 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.758764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.758578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdgg\" (UniqueName: \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.758764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.758615 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.759238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.759211 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.761327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.761303 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.821438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.821378 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdgg\" (UniqueName: \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\") pod \"maas-api-revoke-sub-b8e4a00a-fgdcv\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:18.840394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:18.840308 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:19.223444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:19.223344 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv"] Nov 01 05:19:19.242637 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:19.242581 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3244671b_5d2e_4ed1_a34c_4cb0e40f6734.slice/crio-5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46 WatchSource:0}: Error finding container 5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46: Status 404 returned error can't find the container with id 5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46 Nov 01 05:19:19.932958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:19.932915 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" event={"ID":"3244671b-5d2e-4ed1-a34c-4cb0e40f6734","Type":"ContainerStarted","Data":"ba21f1992f6385a2975234f233e9f11c11eb287afc1c8b47faef2a5db8475bf4"} Nov 01 05:19:19.933396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:19.932988 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" event={"ID":"3244671b-5d2e-4ed1-a34c-4cb0e40f6734","Type":"ContainerStarted","Data":"5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46"} Nov 01 05:19:20.051972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:20.051922 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" podStartSLOduration=2.051904328 podStartE2EDuration="2.051904328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:20.050755513 +0000 UTC m=+975.191344433" watchObservedRunningTime="2026-11-01 05:19:20.051904328 +0000 UTC m=+975.192493247" Nov 01 05:19:20.940757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:20.940720 4425 generic.go:356] "Generic (PLEG): container finished" podID="3244671b-5d2e-4ed1-a34c-4cb0e40f6734" containerID="ba21f1992f6385a2975234f233e9f11c11eb287afc1c8b47faef2a5db8475bf4" exitCode=0 Nov 01 05:19:20.941209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:20.940779 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" event={"ID":"3244671b-5d2e-4ed1-a34c-4cb0e40f6734","Type":"ContainerDied","Data":"ba21f1992f6385a2975234f233e9f11c11eb287afc1c8b47faef2a5db8475bf4"} Nov 01 05:19:22.185348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.185307 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:22.206796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.206689 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\" (UniqueName: \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\") pod \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " Nov 01 05:19:22.206796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.206746 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\") pod \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " Nov 01 05:19:22.207112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.206942 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\") pod \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\" (UID: \"3244671b-5d2e-4ed1-a34c-4cb0e40f6734\") " Nov 01 05:19:22.207991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.207961 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca" pod "3244671b-5d2e-4ed1-a34c-4cb0e40f6734" (UID: "3244671b-5d2e-4ed1-a34c-4cb0e40f6734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:22.224329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.222703 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg" pod "3244671b-5d2e-4ed1-a34c-4cb0e40f6734" (UID: "3244671b-5d2e-4ed1-a34c-4cb0e40f6734"). InnerVolumeSpecName "kube-api-access-svdgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:22.226571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.226538 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca" pod "3244671b-5d2e-4ed1-a34c-4cb0e40f6734" (UID: "3244671b-5d2e-4ed1-a34c-4cb0e40f6734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:22.310463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.310426 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:22.310717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.310696 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdgg\" (UniqueName: \"kubernetes.io/projected/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-kube-api-access-svdgg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:22.310856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.310842 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3244671b-5d2e-4ed1-a34c-4cb0e40f6734-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:22.955091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.955058 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" Nov 01 05:19:22.955315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.955133 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv" event={"ID":"3244671b-5d2e-4ed1-a34c-4cb0e40f6734","Type":"ContainerDied","Data":"5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46"} Nov 01 05:19:22.955315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:22.955224 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eabaa4da83b9afc726f1432151f1d86d1e104b8182b28bea17effe4105f6f46" Nov 01 05:19:26.947450 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:19:26.947411 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:19:29.334240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.334200 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz"] Nov 01 05:19:29.334945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.334924 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3244671b-5d2e-4ed1-a34c-4cb0e40f6734" containerName="revoke-keys" Nov 01 05:19:29.335042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.334948 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3244671b-5d2e-4ed1-a34c-4cb0e40f6734" containerName="revoke-keys" Nov 01 05:19:29.335131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.335118 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3244671b-5d2e-4ed1-a34c-4cb0e40f6734" containerName="revoke-keys" Nov 01 05:19:29.338735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.338713 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.389465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.389432 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.389646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.389503 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jw5g\" (UniqueName: \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.389646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.389597 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.394382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.394200 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:29.398241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.398218 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz"] Nov 01 05:19:29.444954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.444915 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:29.444954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.444942 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:29.444954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.444955 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:29.490794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.490750 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jw5g\" (UniqueName: \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.490794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.490798 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.491051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.490972 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.491663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.491645 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.493381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.493356 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.553024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.552989 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jw5g\" (UniqueName: \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\") pod \"maas-api-revoke-sub-8e18c901-rsqwz\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:29.651780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:29.651702 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:30.031900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:30.031869 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz"] Nov 01 05:19:30.033184 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:30.033156 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637bbb5f_caf3_4736_aaa8_5f91aff1c87b.slice/crio-2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa WatchSource:0}: Error finding container 2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa: Status 404 returned error can't find the container with id 2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa Nov 01 05:19:31.007704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:31.007626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" event={"ID":"637bbb5f-caf3-4736-aaa8-5f91aff1c87b","Type":"ContainerStarted","Data":"1f99a623f654901ce2768c49516b9048e0de28cf6e50195189516e598c1e0c87"} Nov 01 05:19:31.007704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:31.007693 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" event={"ID":"637bbb5f-caf3-4736-aaa8-5f91aff1c87b","Type":"ContainerStarted","Data":"2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa"} Nov 01 05:19:32.018983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:32.018942 4425 generic.go:356] "Generic (PLEG): container finished" podID="637bbb5f-caf3-4736-aaa8-5f91aff1c87b" containerID="1f99a623f654901ce2768c49516b9048e0de28cf6e50195189516e598c1e0c87" exitCode=0 Nov 01 05:19:32.019517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:32.019060 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" event={"ID":"637bbb5f-caf3-4736-aaa8-5f91aff1c87b","Type":"ContainerDied","Data":"1f99a623f654901ce2768c49516b9048e0de28cf6e50195189516e598c1e0c87"} Nov 01 05:19:32.976539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:32.976508 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk"] Nov 01 05:19:32.980603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:32.980586 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.031736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.031694 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.032127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.031858 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.032127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.032078 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm57\" (UniqueName: \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.039080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.039045 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk"] Nov 01 05:19:33.133354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.133311 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.133543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.133431 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm57\" (UniqueName: \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.133543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.133473 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.134001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.133978 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.136098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.136070 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.190019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.189992 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:33.195845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.195819 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm57\" (UniqueName: \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\") pod \"maas-api-revoke-sub-c9be62ec-6b8fk\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.234528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.234452 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\") pod \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " Nov 01 05:19:33.234528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.234520 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\") pod \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " Nov 01 05:19:33.234709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.234624 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\" (UniqueName: \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\") pod \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\" (UID: \"637bbb5f-caf3-4736-aaa8-5f91aff1c87b\") " Nov 01 05:19:33.234904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.234878 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca" pod "637bbb5f-caf3-4736-aaa8-5f91aff1c87b" (UID: "637bbb5f-caf3-4736-aaa8-5f91aff1c87b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:33.235051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.235038 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:33.236879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.236853 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca" pod "637bbb5f-caf3-4736-aaa8-5f91aff1c87b" (UID: "637bbb5f-caf3-4736-aaa8-5f91aff1c87b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:33.236986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.236912 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g" pod "637bbb5f-caf3-4736-aaa8-5f91aff1c87b" (UID: "637bbb5f-caf3-4736-aaa8-5f91aff1c87b"). InnerVolumeSpecName "kube-api-access-8jw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:33.294410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.294378 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:33.336811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.336771 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:33.336811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.336815 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jw5g\" (UniqueName: \"kubernetes.io/projected/637bbb5f-caf3-4736-aaa8-5f91aff1c87b-kube-api-access-8jw5g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:33.673445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:33.673420 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk"] Nov 01 05:19:33.674212 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:33.674189 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c968b76_9af8_4cc8_9d31_2e87dfad9e1d.slice/crio-d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37 WatchSource:0}: Error finding container d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37: Status 404 returned error can't find the container with id d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37 Nov 01 05:19:34.038330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.034008 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" Nov 01 05:19:34.038330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.034034 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz" event={"ID":"637bbb5f-caf3-4736-aaa8-5f91aff1c87b","Type":"ContainerDied","Data":"2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa"} Nov 01 05:19:34.038330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.034158 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f22b39d878c203f82f8fc252c0b45279e4b7347b5b0b1191ea9f4d97bea19aa" Nov 01 05:19:34.038330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.038009 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" event={"ID":"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d","Type":"ContainerStarted","Data":"be180caef74c8749a8787dd0f98dfffba9af281fa15715b6c3d13ecbdf219ce3"} Nov 01 05:19:34.038330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.038055 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" event={"ID":"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d","Type":"ContainerStarted","Data":"d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37"} Nov 01 05:19:34.165424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:34.158799 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" podStartSLOduration=2.158781774 podStartE2EDuration="2.158781774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:34.155503184 +0000 UTC m=+989.296092128" watchObservedRunningTime="2026-11-01 05:19:34.158781774 +0000 UTC m=+989.299370694" Nov 01 05:19:35.046462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:35.046431 4425 generic.go:356] "Generic (PLEG): container finished" podID="0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" containerID="be180caef74c8749a8787dd0f98dfffba9af281fa15715b6c3d13ecbdf219ce3" exitCode=0 Nov 01 05:19:35.046873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:35.046493 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" event={"ID":"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d","Type":"ContainerDied","Data":"be180caef74c8749a8787dd0f98dfffba9af281fa15715b6c3d13ecbdf219ce3"} Nov 01 05:19:36.243227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.243189 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:36.372662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.372626 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\" (UniqueName: \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\") pod \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " Nov 01 05:19:36.372867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.372697 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\") pod \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " Nov 01 05:19:36.372867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.372732 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\") pod \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\" (UID: \"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d\") " Nov 01 05:19:36.373166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.373135 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca" pod "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" (UID: "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:36.375178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.375146 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca" pod "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" (UID: "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:36.375511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.375491 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57" pod "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" (UID: "0c968b76-9af8-4cc8-9d31-2e87dfad9e1d"). InnerVolumeSpecName "kube-api-access-7jm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:36.474430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.474391 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:36.474430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.474429 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jm57\" (UniqueName: \"kubernetes.io/projected/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-kube-api-access-7jm57\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:36.474696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:36.474444 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:37.066763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:37.066718 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" event={"ID":"0c968b76-9af8-4cc8-9d31-2e87dfad9e1d","Type":"ContainerDied","Data":"d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37"} Nov 01 05:19:37.066957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:37.066792 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d071141f9cfd76c9ccbe05782d5057389233ea1d45a0b8b17f6f54fcf0157d37" Nov 01 05:19:37.066957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:37.066872 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk" Nov 01 05:19:43.783851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.783821 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh"] Nov 01 05:19:43.784663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784644 4425 cpu_manager.go:414] "containerMap: removing container" podUID="637bbb5f-caf3-4736-aaa8-5f91aff1c87b" containerName="revoke-keys" Nov 01 05:19:43.784711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784667 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="637bbb5f-caf3-4736-aaa8-5f91aff1c87b" containerName="revoke-keys" Nov 01 05:19:43.784745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784733 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" containerName="revoke-keys" Nov 01 05:19:43.784745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784741 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" containerName="revoke-keys" Nov 01 05:19:43.784873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784861 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" containerName="revoke-keys" Nov 01 05:19:43.784919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.784879 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="637bbb5f-caf3-4736-aaa8-5f91aff1c87b" containerName="revoke-keys" Nov 01 05:19:43.789134 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.789106 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.843958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.843896 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh"] Nov 01 05:19:43.866827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.866783 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.867035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.866858 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.867035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.866927 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4lv\" (UniqueName: \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.890963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.890934 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:43.890963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.890952 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:43.891167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.890961 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:43.893908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.893889 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:43.968177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.968143 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.968177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.968176 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4lv\" (UniqueName: \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.968405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.968332 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.968941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.968916 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:43.970949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:43.970926 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:44.030920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:44.030889 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4lv\" (UniqueName: \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\") pod \"maas-api-revoke-sub-e3910eaf-sqkvh\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:44.103220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:44.103184 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:44.482938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:44.482915 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh"] Nov 01 05:19:44.484138 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:44.484105 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeda527b_a7c9_4ee0_9add_1bad810a2c75.slice/crio-18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8 WatchSource:0}: Error finding container 18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8: Status 404 returned error can't find the container with id 18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8 Nov 01 05:19:45.123873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.123832 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" event={"ID":"beda527b-a7c9-4ee0-9add-1bad810a2c75","Type":"ContainerStarted","Data":"3041f0e6e24325747ad3343661d7111a6a1600c2bd5ff171eb4f4a779e47f05b"} Nov 01 05:19:45.124306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.123902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" event={"ID":"beda527b-a7c9-4ee0-9add-1bad810a2c75","Type":"ContainerStarted","Data":"18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8"} Nov 01 05:19:45.245291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.245225 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" podStartSLOduration=2.245211403 podStartE2EDuration="2.245211403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:45.243818195 +0000 UTC m=+1000.384407126" watchObservedRunningTime="2026-11-01 05:19:45.245211403 +0000 UTC m=+1000.385800328" Nov 01 05:19:45.870750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.870718 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58"] Nov 01 05:19:45.874721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.874706 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:45.933232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.933200 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58"] Nov 01 05:19:45.997679 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.997651 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:45.997845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.997799 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bk4x\" (UniqueName: \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:45.997886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:45.997847 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.098808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.098767 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bk4x\" (UniqueName: \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.099004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.098824 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.099004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.098929 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.099683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.099656 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.101636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.101612 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.131540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.131455 4425 generic.go:356] "Generic (PLEG): container finished" podID="beda527b-a7c9-4ee0-9add-1bad810a2c75" containerID="3041f0e6e24325747ad3343661d7111a6a1600c2bd5ff171eb4f4a779e47f05b" exitCode=0 Nov 01 05:19:46.131919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.131541 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" event={"ID":"beda527b-a7c9-4ee0-9add-1bad810a2c75","Type":"ContainerDied","Data":"3041f0e6e24325747ad3343661d7111a6a1600c2bd5ff171eb4f4a779e47f05b"} Nov 01 05:19:46.161852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.161824 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bk4x\" (UniqueName: \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\") pod \"maas-api-revoke-sub-ab32cbec-48h58\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.188677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.188640 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:46.571729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:46.571700 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58"] Nov 01 05:19:46.571869 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:46.571842 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf700d3cf_b14f_4a0d_ba51_7072244c0873.slice/crio-dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f WatchSource:0}: Error finding container dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f: Status 404 returned error can't find the container with id dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f Nov 01 05:19:47.139718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.139627 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" event={"ID":"f700d3cf-b14f-4a0d-ba51-7072244c0873","Type":"ContainerStarted","Data":"58c44d9cc71d8e46176658f746ddd512c8f9065bdb855d629f1238e56a257c28"} Nov 01 05:19:47.139718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.139706 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" event={"ID":"f700d3cf-b14f-4a0d-ba51-7072244c0873","Type":"ContainerStarted","Data":"dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f"} Nov 01 05:19:47.259000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.258949 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" podStartSLOduration=2.258935512 podStartE2EDuration="2.258935512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:47.257497757 +0000 UTC m=+1002.398086671" watchObservedRunningTime="2026-11-01 05:19:47.258935512 +0000 UTC m=+1002.399524431" Nov 01 05:19:47.292951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.292923 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:47.416437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.416354 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\") pod \"beda527b-a7c9-4ee0-9add-1bad810a2c75\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " Nov 01 05:19:47.416617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.416502 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\") pod \"beda527b-a7c9-4ee0-9add-1bad810a2c75\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " Nov 01 05:19:47.416617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.416541 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\" (UniqueName: \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\") pod \"beda527b-a7c9-4ee0-9add-1bad810a2c75\" (UID: \"beda527b-a7c9-4ee0-9add-1bad810a2c75\") " Nov 01 05:19:47.419307 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.418127 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca" pod "beda527b-a7c9-4ee0-9add-1bad810a2c75" (UID: "beda527b-a7c9-4ee0-9add-1bad810a2c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:47.437493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.419752 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca" pod "beda527b-a7c9-4ee0-9add-1bad810a2c75" (UID: "beda527b-a7c9-4ee0-9add-1bad810a2c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:47.437493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.420101 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv" pod "beda527b-a7c9-4ee0-9add-1bad810a2c75" (UID: "beda527b-a7c9-4ee0-9add-1bad810a2c75"). InnerVolumeSpecName "kube-api-access-gd4lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:47.518463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.518422 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:47.518463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.518459 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beda527b-a7c9-4ee0-9add-1bad810a2c75-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:47.518672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:47.518473 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4lv\" (UniqueName: \"kubernetes.io/projected/beda527b-a7c9-4ee0-9add-1bad810a2c75-kube-api-access-gd4lv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:48.147808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:48.147781 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" Nov 01 05:19:48.148298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:48.147807 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh" event={"ID":"beda527b-a7c9-4ee0-9add-1bad810a2c75","Type":"ContainerDied","Data":"18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8"} Nov 01 05:19:48.148298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:48.147858 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18261d93afa54992cd654ebd0bfe3aa4e77fb39dc20460c24795d42658c4e7d8" Nov 01 05:19:48.149590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:48.149562 4425 generic.go:356] "Generic (PLEG): container finished" podID="f700d3cf-b14f-4a0d-ba51-7072244c0873" containerID="58c44d9cc71d8e46176658f746ddd512c8f9065bdb855d629f1238e56a257c28" exitCode=0 Nov 01 05:19:48.149785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:48.149611 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" event={"ID":"f700d3cf-b14f-4a0d-ba51-7072244c0873","Type":"ContainerDied","Data":"58c44d9cc71d8e46176658f746ddd512c8f9065bdb855d629f1238e56a257c28"} Nov 01 05:19:49.306828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.306801 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:49.340092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.340058 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\" (UniqueName: \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\") pod \"f700d3cf-b14f-4a0d-ba51-7072244c0873\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " Nov 01 05:19:49.340318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.340104 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\") pod \"f700d3cf-b14f-4a0d-ba51-7072244c0873\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " Nov 01 05:19:49.340318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.340212 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\") pod \"f700d3cf-b14f-4a0d-ba51-7072244c0873\" (UID: \"f700d3cf-b14f-4a0d-ba51-7072244c0873\") " Nov 01 05:19:49.340602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.340579 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca" pod "f700d3cf-b14f-4a0d-ba51-7072244c0873" (UID: "f700d3cf-b14f-4a0d-ba51-7072244c0873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:49.340720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.340705 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:49.342460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.342436 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca" pod "f700d3cf-b14f-4a0d-ba51-7072244c0873" (UID: "f700d3cf-b14f-4a0d-ba51-7072244c0873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:49.342568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.342518 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x" pod "f700d3cf-b14f-4a0d-ba51-7072244c0873" (UID: "f700d3cf-b14f-4a0d-ba51-7072244c0873"). InnerVolumeSpecName "kube-api-access-2bk4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:49.441627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.441544 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f700d3cf-b14f-4a0d-ba51-7072244c0873-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:49.441627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:49.441584 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bk4x\" (UniqueName: \"kubernetes.io/projected/f700d3cf-b14f-4a0d-ba51-7072244c0873-kube-api-access-2bk4x\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:50.163452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:50.163415 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" Nov 01 05:19:50.163452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:50.163426 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58" event={"ID":"f700d3cf-b14f-4a0d-ba51-7072244c0873","Type":"ContainerDied","Data":"dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f"} Nov 01 05:19:50.163664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:50.163480 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc476ead1ade2b0f59aa6aa2896f90d01debd47ac9058e21cfad340a0033677f" Nov 01 05:19:53.666982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.666949 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76"] Nov 01 05:19:53.667547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667532 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f700d3cf-b14f-4a0d-ba51-7072244c0873" containerName="revoke-keys" Nov 01 05:19:53.667591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667548 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f700d3cf-b14f-4a0d-ba51-7072244c0873" containerName="revoke-keys" Nov 01 05:19:53.667591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667565 4425 cpu_manager.go:414] "containerMap: removing container" podUID="beda527b-a7c9-4ee0-9add-1bad810a2c75" containerName="revoke-keys" Nov 01 05:19:53.667591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667571 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beda527b-a7c9-4ee0-9add-1bad810a2c75" containerName="revoke-keys" Nov 01 05:19:53.667687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667674 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f700d3cf-b14f-4a0d-ba51-7072244c0873" containerName="revoke-keys" Nov 01 05:19:53.667687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.667683 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="beda527b-a7c9-4ee0-9add-1bad810a2c75" containerName="revoke-keys" Nov 01 05:19:53.671175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.671153 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.695963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.695935 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.696109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.695987 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.696163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.696114 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncds\" (UniqueName: \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.728696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.728663 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76"] Nov 01 05:19:53.775986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.775952 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:19:53.775986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.775993 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:19:53.776191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.775993 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:19:53.776191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.776045 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:19:53.797498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.797461 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncds\" (UniqueName: \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.797676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.797599 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.797676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.797647 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.798339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.798316 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.800093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.800075 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.860589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.860560 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncds\" (UniqueName: \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\") pod \"maas-api-revoke-sub-fc1a15be-jpt76\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:53.984904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:53.984819 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:54.363357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:54.363329 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76"] Nov 01 05:19:54.363357 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:54.363343 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c953b90_60f1_476a_a601_d3faa74df00e.slice/crio-4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188 WatchSource:0}: Error finding container 4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188: Status 404 returned error can't find the container with id 4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188 Nov 01 05:19:55.197493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:55.197452 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" event={"ID":"1c953b90-60f1-476a-a601-d3faa74df00e","Type":"ContainerStarted","Data":"3601269034a1b78c1760d73bc6d8f7f4981044ed184ef029678ec3c0415c6a99"} Nov 01 05:19:55.197968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:55.197526 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" event={"ID":"1c953b90-60f1-476a-a601-d3faa74df00e","Type":"ContainerStarted","Data":"4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188"} Nov 01 05:19:55.318603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:55.318555 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" podStartSLOduration=2.318541877 podStartE2EDuration="2.318541877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:55.315989713 +0000 UTC m=+1010.456578636" watchObservedRunningTime="2026-11-01 05:19:55.318541877 +0000 UTC m=+1010.459130796" Nov 01 05:19:56.205457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.205420 4425 generic.go:356] "Generic (PLEG): container finished" podID="1c953b90-60f1-476a-a601-d3faa74df00e" containerID="3601269034a1b78c1760d73bc6d8f7f4981044ed184ef029678ec3c0415c6a99" exitCode=0 Nov 01 05:19:56.205853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.205500 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" event={"ID":"1c953b90-60f1-476a-a601-d3faa74df00e","Type":"ContainerDied","Data":"3601269034a1b78c1760d73bc6d8f7f4981044ed184ef029678ec3c0415c6a99"} Nov 01 05:19:56.687024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.686987 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr"] Nov 01 05:19:56.691149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.691126 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.733431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.733397 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.733613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.733447 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.733613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.733464 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275df\" (UniqueName: \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.749346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.749316 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr"] Nov 01 05:19:56.835328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.835292 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.835506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.835353 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.835506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.835384 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275df\" (UniqueName: \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.836106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.836072 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.837843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.837825 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.898411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:56.898380 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275df\" (UniqueName: \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\") pod \"maas-api-revoke-sub-b5861eed-qxqtr\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:56.948306 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:19:56.948208 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:19:57.005449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.005421 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:19:57.365555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.365531 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:57.382547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.382520 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr"] Nov 01 05:19:57.382859 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:19:57.382838 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5cedc8_5b35_467f_842f_23851e29a91b.slice/crio-25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e WatchSource:0}: Error finding container 25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e: Status 404 returned error can't find the container with id 25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e Nov 01 05:19:57.441545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.441511 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\") pod \"1c953b90-60f1-476a-a601-d3faa74df00e\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " Nov 01 05:19:57.441728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.441604 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\") pod \"1c953b90-60f1-476a-a601-d3faa74df00e\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " Nov 01 05:19:57.441728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.441675 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\" (UniqueName: \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\") pod \"1c953b90-60f1-476a-a601-d3faa74df00e\" (UID: \"1c953b90-60f1-476a-a601-d3faa74df00e\") " Nov 01 05:19:57.441965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.441935 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca" pod "1c953b90-60f1-476a-a601-d3faa74df00e" (UID: "1c953b90-60f1-476a-a601-d3faa74df00e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:19:57.442167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.442144 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:57.444472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.444450 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds" pod "1c953b90-60f1-476a-a601-d3faa74df00e" (UID: "1c953b90-60f1-476a-a601-d3faa74df00e"). InnerVolumeSpecName "kube-api-access-sncds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:19:57.444472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.444458 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca" pod "1c953b90-60f1-476a-a601-d3faa74df00e" (UID: "1c953b90-60f1-476a-a601-d3faa74df00e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:19:57.543522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.543440 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c953b90-60f1-476a-a601-d3faa74df00e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:57.543522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:57.543472 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sncds\" (UniqueName: \"kubernetes.io/projected/1c953b90-60f1-476a-a601-d3faa74df00e-kube-api-access-sncds\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:19:58.226754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.226644 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" event={"ID":"1f5cedc8-5b35-467f-842f-23851e29a91b","Type":"ContainerStarted","Data":"6cabbc5f7c99edeeb5c6f6ca96a163b43f490609728e843e7a28fe444aa2f2d6"} Nov 01 05:19:58.226754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.226741 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" event={"ID":"1f5cedc8-5b35-467f-842f-23851e29a91b","Type":"ContainerStarted","Data":"25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e"} Nov 01 05:19:58.228200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.228176 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" Nov 01 05:19:58.228330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.228210 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76" event={"ID":"1c953b90-60f1-476a-a601-d3faa74df00e","Type":"ContainerDied","Data":"4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188"} Nov 01 05:19:58.228330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.228276 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa6bbd673b157333466051504c59036310b3cd0980cff7d6c8169076f13c188" Nov 01 05:19:58.344968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:58.344914 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" podStartSLOduration=2.344901283 podStartE2EDuration="2.344901283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:58.343903018 +0000 UTC m=+1013.484491960" watchObservedRunningTime="2026-11-01 05:19:58.344901283 +0000 UTC m=+1013.485490201" Nov 01 05:19:59.236108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:59.236070 4425 generic.go:356] "Generic (PLEG): container finished" podID="1f5cedc8-5b35-467f-842f-23851e29a91b" containerID="6cabbc5f7c99edeeb5c6f6ca96a163b43f490609728e843e7a28fe444aa2f2d6" exitCode=0 Nov 01 05:19:59.236532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:19:59.236147 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" event={"ID":"1f5cedc8-5b35-467f-842f-23851e29a91b","Type":"ContainerDied","Data":"6cabbc5f7c99edeeb5c6f6ca96a163b43f490609728e843e7a28fe444aa2f2d6"} Nov 01 05:20:00.394438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.394413 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:20:00.488851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.488817 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\") pod \"1f5cedc8-5b35-467f-842f-23851e29a91b\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " Nov 01 05:20:00.489052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.488859 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\") pod \"1f5cedc8-5b35-467f-842f-23851e29a91b\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " Nov 01 05:20:00.489052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.489022 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\" (UniqueName: \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\") pod \"1f5cedc8-5b35-467f-842f-23851e29a91b\" (UID: \"1f5cedc8-5b35-467f-842f-23851e29a91b\") " Nov 01 05:20:00.489286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.489224 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca" pod "1f5cedc8-5b35-467f-842f-23851e29a91b" (UID: "1f5cedc8-5b35-467f-842f-23851e29a91b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:00.489636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.489606 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:00.491246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.491218 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca" pod "1f5cedc8-5b35-467f-842f-23851e29a91b" (UID: "1f5cedc8-5b35-467f-842f-23851e29a91b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:00.491413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.491393 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df" pod "1f5cedc8-5b35-467f-842f-23851e29a91b" (UID: "1f5cedc8-5b35-467f-842f-23851e29a91b"). InnerVolumeSpecName "kube-api-access-275df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:00.591122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.591075 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275df\" (UniqueName: \"kubernetes.io/projected/1f5cedc8-5b35-467f-842f-23851e29a91b-kube-api-access-275df\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:00.591122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:00.591113 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f5cedc8-5b35-467f-842f-23851e29a91b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:01.253359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:01.253238 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" Nov 01 05:20:01.253359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:01.253291 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr" event={"ID":"1f5cedc8-5b35-467f-842f-23851e29a91b","Type":"ContainerDied","Data":"25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e"} Nov 01 05:20:01.253359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:01.253341 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25180bc487f6551f9b3c6d93a5a8d0251bc0775d2063734234879089bd27ee2e" Nov 01 05:20:03.353631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.353598 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:20:03.354376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354355 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1c953b90-60f1-476a-a601-d3faa74df00e" containerName="revoke-keys" Nov 01 05:20:03.354376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354376 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c953b90-60f1-476a-a601-d3faa74df00e" containerName="revoke-keys" Nov 01 05:20:03.354577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354408 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1f5cedc8-5b35-467f-842f-23851e29a91b" containerName="revoke-keys" Nov 01 05:20:03.354577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354415 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5cedc8-5b35-467f-842f-23851e29a91b" containerName="revoke-keys" Nov 01 05:20:03.354577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354565 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c953b90-60f1-476a-a601-d3faa74df00e" containerName="revoke-keys" Nov 01 05:20:03.354747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.354585 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5cedc8-5b35-467f-842f-23851e29a91b" containerName="revoke-keys" Nov 01 05:20:03.358321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.358300 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.434803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.416904 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2864b6-openshift-default-dockercfg-nw7w6\"" Nov 01 05:20:03.441117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441074 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441141 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441237 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441419 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441457 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441489 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441516 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441574 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqdx\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.441681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.441634 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.473537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.473492 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:20:03.543268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543218 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqdx\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543333 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543440 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543469 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543507 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543630 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543662 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543682 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.543778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.543706 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.544071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.544043 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.544305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.544225 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.544305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.544096 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.544305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.544195 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.544554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.544479 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.546231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.546204 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.546683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.546652 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.605800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.605728 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqdx\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.605800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.605725 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\") pod \"e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:03.672355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:03.672157 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:04.137659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.137626 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:20:04.138504 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:04.138469 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa71ec23_30b9_460c_b9b6_e054d0640203.slice/crio-3e625a11b5907f38c2439dd6ad75113fb99084ab363a5593a504b7eec05f16e4 WatchSource:0}: Error finding container 3e625a11b5907f38c2439dd6ad75113fb99084ab363a5593a504b7eec05f16e4: Status 404 returned error can't find the container with id 3e625a11b5907f38c2439dd6ad75113fb99084ab363a5593a504b7eec05f16e4 Nov 01 05:20:04.140958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.140921 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:04.141061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.140993 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:04.141061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.141027 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:04.326299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.319305 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerStarted","Data":"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1"} Nov 01 05:20:04.326299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.319374 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerStarted","Data":"3e625a11b5907f38c2439dd6ad75113fb99084ab363a5593a504b7eec05f16e4"} Nov 01 05:20:04.698294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:04.673465 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:05.680810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:05.680778 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:05.920300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:05.920219 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podStartSLOduration=2.920202119 podStartE2EDuration="2.920202119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:04.463698149 +0000 UTC m=+1019.604287069" watchObservedRunningTime="2026-11-01 05:20:05.920202119 +0000 UTC m=+1021.060791041" Nov 01 05:20:06.331981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:06.331943 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:06.332852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:06.332831 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:20:10.034235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.034199 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr"] Nov 01 05:20:10.038704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.038687 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.095612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.095585 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr"] Nov 01 05:20:10.142682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.142648 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:20:10.142682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.142676 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:20:10.142886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.142741 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:20:10.143625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.143608 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:20:10.158136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.158113 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb64c\" (UniqueName: \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.158295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.158240 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.158346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.158309 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.259061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.259023 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb64c\" (UniqueName: \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.259246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.259125 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.259246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.259152 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.259784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.259762 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.261693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.261669 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.321424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.321397 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb64c\" (UniqueName: \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\") pod \"maas-api-revoke-sub-43004b85-zdsgr\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.351856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.351830 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:10.731450 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:10.730822 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9225c985_39b2_4c44_b73f_858c14fda99d.slice/crio-2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a WatchSource:0}: Error finding container 2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a: Status 404 returned error can't find the container with id 2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a Nov 01 05:20:10.731947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:10.731921 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr"] Nov 01 05:20:11.365746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:11.365708 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" event={"ID":"9225c985-39b2-4c44-b73f-858c14fda99d","Type":"ContainerStarted","Data":"cc95224ec5b7fd15e4f9268d7ce0c98da6b1388361c0e64ea1ae0ed5e51f609a"} Nov 01 05:20:11.366124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:11.365769 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" event={"ID":"9225c985-39b2-4c44-b73f-858c14fda99d","Type":"ContainerStarted","Data":"2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a"} Nov 01 05:20:11.538102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:11.538049 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" podStartSLOduration=2.538034434 podStartE2EDuration="2.538034434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:11.483835806 +0000 UTC m=+1026.624424736" watchObservedRunningTime="2026-11-01 05:20:11.538034434 +0000 UTC m=+1026.678623353" Nov 01 05:20:12.373886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:12.373849 4425 generic.go:356] "Generic (PLEG): container finished" podID="9225c985-39b2-4c44-b73f-858c14fda99d" containerID="cc95224ec5b7fd15e4f9268d7ce0c98da6b1388361c0e64ea1ae0ed5e51f609a" exitCode=0 Nov 01 05:20:12.374291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:12.373934 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" event={"ID":"9225c985-39b2-4c44-b73f-858c14fda99d","Type":"ContainerDied","Data":"cc95224ec5b7fd15e4f9268d7ce0c98da6b1388361c0e64ea1ae0ed5e51f609a"} Nov 01 05:20:13.550364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.550338 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:13.717502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.717419 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\") pod \"9225c985-39b2-4c44-b73f-858c14fda99d\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " Nov 01 05:20:13.717502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.717477 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\" (UniqueName: \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\") pod \"9225c985-39b2-4c44-b73f-858c14fda99d\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " Nov 01 05:20:13.717718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.717557 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\") pod \"9225c985-39b2-4c44-b73f-858c14fda99d\" (UID: \"9225c985-39b2-4c44-b73f-858c14fda99d\") " Nov 01 05:20:13.717838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.717813 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca" pod "9225c985-39b2-4c44-b73f-858c14fda99d" (UID: "9225c985-39b2-4c44-b73f-858c14fda99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:13.718039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.718024 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:13.719685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.719663 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c" pod "9225c985-39b2-4c44-b73f-858c14fda99d" (UID: "9225c985-39b2-4c44-b73f-858c14fda99d"). InnerVolumeSpecName "kube-api-access-xb64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:13.719762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.719725 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca" pod "9225c985-39b2-4c44-b73f-858c14fda99d" (UID: "9225c985-39b2-4c44-b73f-858c14fda99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:13.823096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.823062 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb64c\" (UniqueName: \"kubernetes.io/projected/9225c985-39b2-4c44-b73f-858c14fda99d-kube-api-access-xb64c\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:13.823375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.823098 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9225c985-39b2-4c44-b73f-858c14fda99d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:13.868348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.868308 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb"] Nov 01 05:20:13.868928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.868908 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9225c985-39b2-4c44-b73f-858c14fda99d" containerName="revoke-keys" Nov 01 05:20:13.869027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.868929 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9225c985-39b2-4c44-b73f-858c14fda99d" containerName="revoke-keys" Nov 01 05:20:13.869116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.869103 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9225c985-39b2-4c44-b73f-858c14fda99d" containerName="revoke-keys" Nov 01 05:20:13.872603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.872581 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:13.929960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:13.929929 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb"] Nov 01 05:20:14.025982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.025867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.025982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.025951 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rlj7\" (UniqueName: \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.026211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.026155 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.127685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.127638 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.127986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.127953 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rlj7\" (UniqueName: \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.128246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.128218 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.128402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.128382 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.130858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.130834 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.191945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.191908 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rlj7\" (UniqueName: \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\") pod \"maas-api-revoke-sub-3d8a126a-m58bb\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.269299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.267554 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:20:14.272524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.272359 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.324703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.324668 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:20:14.326274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.326218 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2864b6\"" Nov 01 05:20:14.326421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.326243 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2864b6-dockercfg-nv59s\"" Nov 01 05:20:14.355159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.355120 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.355339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.355263 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxtw\" (UniqueName: \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.389727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.389686 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" event={"ID":"9225c985-39b2-4c44-b73f-858c14fda99d","Type":"ContainerDied","Data":"2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a"} Nov 01 05:20:14.389921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.389743 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1cbb1a1b805636afbd38e56817b2dc910917791fd59630df063b5da2bdc23a" Nov 01 05:20:14.389921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.389765 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr" Nov 01 05:20:14.456223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.456190 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.456413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.456275 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxtw\" (UniqueName: \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.456896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.456875 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.486778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.486746 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:14.519293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.519268 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxtw\" (UniqueName: \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\") pod \"payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.589039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.589008 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:14.864304 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:14.864272 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816d4584_91b0_4f65_8902_30f6acdbd18c.slice/crio-bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226 WatchSource:0}: Error finding container bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226: Status 404 returned error can't find the container with id bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226 Nov 01 05:20:14.864451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.864409 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb"] Nov 01 05:20:14.973561 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:14.973536 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64641330_77d1_44af_b67d_c140db584523.slice/crio-97c77b5e16d6f79a9e4b9c229f7c459ff14069157d3fa2af2b57fd1fae615cfe WatchSource:0}: Error finding container 97c77b5e16d6f79a9e4b9c229f7c459ff14069157d3fa2af2b57fd1fae615cfe: Status 404 returned error can't find the container with id 97c77b5e16d6f79a9e4b9c229f7c459ff14069157d3fa2af2b57fd1fae615cfe Nov 01 05:20:14.974566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:14.974548 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:20:15.397518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.397481 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" event={"ID":"64641330-77d1-44af-b67d-c140db584523","Type":"ContainerStarted","Data":"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7"} Nov 01 05:20:15.397705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.397533 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" event={"ID":"64641330-77d1-44af-b67d-c140db584523","Type":"ContainerStarted","Data":"97c77b5e16d6f79a9e4b9c229f7c459ff14069157d3fa2af2b57fd1fae615cfe"} Nov 01 05:20:15.397705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.397690 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:15.399343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.399318 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" event={"ID":"816d4584-91b0-4f65-8902-30f6acdbd18c","Type":"ContainerStarted","Data":"6fa1a6f0d26f756f4746873727091b7145dc758820f5b0a538b30b455f8c645b"} Nov 01 05:20:15.399458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.399359 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" event={"ID":"816d4584-91b0-4f65-8902-30f6acdbd18c","Type":"ContainerStarted","Data":"bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226"} Nov 01 05:20:15.521057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.520959 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" podStartSLOduration=1.520945338 podStartE2EDuration="1.520945338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:15.51630958 +0000 UTC m=+1030.656898495" watchObservedRunningTime="2026-11-01 05:20:15.520945338 +0000 UTC m=+1030.661534304" Nov 01 05:20:15.634679 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:15.634627 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" podStartSLOduration=2.634612499 podStartE2EDuration="2.634612499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:15.634229938 +0000 UTC m=+1030.774818858" watchObservedRunningTime="2026-11-01 05:20:15.634612499 +0000 UTC m=+1030.775201419" Nov 01 05:20:16.408244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:16.408163 4425 generic.go:356] "Generic (PLEG): container finished" podID="816d4584-91b0-4f65-8902-30f6acdbd18c" containerID="6fa1a6f0d26f756f4746873727091b7145dc758820f5b0a538b30b455f8c645b" exitCode=0 Nov 01 05:20:16.408435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:16.408245 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" event={"ID":"816d4584-91b0-4f65-8902-30f6acdbd18c","Type":"ContainerDied","Data":"6fa1a6f0d26f756f4746873727091b7145dc758820f5b0a538b30b455f8c645b"} Nov 01 05:20:17.574855 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.574824 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:17.591072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.591043 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\" (UniqueName: \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\") pod \"816d4584-91b0-4f65-8902-30f6acdbd18c\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " Nov 01 05:20:17.591240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.591100 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\") pod \"816d4584-91b0-4f65-8902-30f6acdbd18c\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " Nov 01 05:20:17.591883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.591284 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\") pod \"816d4584-91b0-4f65-8902-30f6acdbd18c\" (UID: \"816d4584-91b0-4f65-8902-30f6acdbd18c\") " Nov 01 05:20:17.591883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.591482 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca" pod "816d4584-91b0-4f65-8902-30f6acdbd18c" (UID: "816d4584-91b0-4f65-8902-30f6acdbd18c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:17.591883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.591648 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:17.593581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.593550 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca" pod "816d4584-91b0-4f65-8902-30f6acdbd18c" (UID: "816d4584-91b0-4f65-8902-30f6acdbd18c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:17.593673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.593550 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7" pod "816d4584-91b0-4f65-8902-30f6acdbd18c" (UID: "816d4584-91b0-4f65-8902-30f6acdbd18c"). InnerVolumeSpecName "kube-api-access-9rlj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:17.693410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.693313 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d4584-91b0-4f65-8902-30f6acdbd18c-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:17.693410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:17.693349 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rlj7\" (UniqueName: \"kubernetes.io/projected/816d4584-91b0-4f65-8902-30f6acdbd18c-kube-api-access-9rlj7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:18.422689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:18.422660 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" Nov 01 05:20:18.422897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:18.422663 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb" event={"ID":"816d4584-91b0-4f65-8902-30f6acdbd18c","Type":"ContainerDied","Data":"bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226"} Nov 01 05:20:18.422897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:18.422786 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd78372be2536946fccc7279eaee1ed89a9f3a9e23b7303f230a1be2b6e8226" Nov 01 05:20:24.535695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.535665 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:20:24.536276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.536239 4425 cpu_manager.go:414] "containerMap: removing container" podUID="816d4584-91b0-4f65-8902-30f6acdbd18c" containerName="revoke-keys" Nov 01 05:20:24.536276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.536267 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d4584-91b0-4f65-8902-30f6acdbd18c" containerName="revoke-keys" Nov 01 05:20:24.536396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.536385 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d4584-91b0-4f65-8902-30f6acdbd18c" containerName="revoke-keys" Nov 01 05:20:24.540308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.540289 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.583900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.583863 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.583922 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584048 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584082 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584214 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584437 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqdn\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584515 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.584599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.584584 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.651797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.651757 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:20:24.685548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685513 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685562 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685592 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685606 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685757 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685856 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685891 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqdn\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685914 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.685992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685977 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.686102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.686070 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.686154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.685984 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.686317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.686300 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.686317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.686309 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.686423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.686410 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.688344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.688314 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.688617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.688597 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.748262 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.748232 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.748443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.748394 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqdn\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:24.853997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:24.853965 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:25.290146 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:25.290104 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457dd741_95f2_45ed_89f7_5a6dcafb151d.slice/crio-74fb3a2a74ae30b7b20ea558d3b67619519978adccf0c11e18f5c0b5f56d49d3 WatchSource:0}: Error finding container 74fb3a2a74ae30b7b20ea558d3b67619519978adccf0c11e18f5c0b5f56d49d3: Status 404 returned error can't find the container with id 74fb3a2a74ae30b7b20ea558d3b67619519978adccf0c11e18f5c0b5f56d49d3 Nov 01 05:20:25.290695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.290666 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:20:25.293376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.293334 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:25.293459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.293435 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:25.293504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.293480 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:20:25.477710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.477676 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" event={"ID":"457dd741-95f2-45ed-89f7-5a6dcafb151d","Type":"ContainerStarted","Data":"18633dda041990b9ada7051cd4ba2ea7b9906c5d60047214f872b9fe4578a6d0"} Nov 01 05:20:25.477868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.477733 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" event={"ID":"457dd741-95f2-45ed-89f7-5a6dcafb151d","Type":"ContainerStarted","Data":"74fb3a2a74ae30b7b20ea558d3b67619519978adccf0c11e18f5c0b5f56d49d3"} Nov 01 05:20:25.604735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.604675 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podStartSLOduration=1.6046574919999999 podStartE2EDuration="1.604657492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:25.600760131 +0000 UTC m=+1040.741349071" watchObservedRunningTime="2026-11-01 05:20:25.604657492 +0000 UTC m=+1040.745246416" Nov 01 05:20:25.655442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.655411 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf"] Nov 01 05:20:25.659875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.659852 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.715904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.715876 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:20:25.716117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.715981 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:20:25.716117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.715988 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:20:25.716234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.716038 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:20:25.716543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.716524 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf"] Nov 01 05:20:25.799238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.799200 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.799715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.799693 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.799902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.799884 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwclx\" (UniqueName: \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.862464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.862358 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:25.900742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.900706 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.900742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.900745 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwclx\" (UniqueName: \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.900955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.900802 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.901533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.901499 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.903173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.903150 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.969694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.969650 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwclx\" (UniqueName: \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\") pod \"maas-api-revoke-sub-30176787-twnmf\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:25.975421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:25.975399 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:26.368837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:26.368796 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf"] Nov 01 05:20:26.370648 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:26.370625 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed46963_d5f2_4d87_be6a_290657ef88da.slice/crio-dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2 WatchSource:0}: Error finding container dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2: Status 404 returned error can't find the container with id dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2 Nov 01 05:20:26.412308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:26.412282 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:20:26.486228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:26.486189 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" event={"ID":"3ed46963-d5f2-4d87-be6a-290657ef88da","Type":"ContainerStarted","Data":"dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2"} Nov 01 05:20:26.866607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:26.866571 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 05:20:26.867243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:26.867209 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 05:20:26.974416 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:20:26.974378 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:20:27.494809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:27.494774 4425 generic.go:356] "Generic (PLEG): container finished" podID="3ed46963-d5f2-4d87-be6a-290657ef88da" containerID="c665cb831c12b7eb78612320581cd435fe6b18e3b92852f9fa5a4ab459869933" exitCode=0 Nov 01 05:20:27.495000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:27.494867 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" event={"ID":"3ed46963-d5f2-4d87-be6a-290657ef88da","Type":"ContainerDied","Data":"c665cb831c12b7eb78612320581cd435fe6b18e3b92852f9fa5a4ab459869933"} Nov 01 05:20:27.857752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:27.857712 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 05:20:27.858054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:27.858026 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 05:20:28.695496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.695467 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:28.830810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.830771 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\") pod \"3ed46963-d5f2-4d87-be6a-290657ef88da\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " Nov 01 05:20:28.831028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.830996 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\" (UniqueName: \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\") pod \"3ed46963-d5f2-4d87-be6a-290657ef88da\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " Nov 01 05:20:28.831110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.831086 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca" pod "3ed46963-d5f2-4d87-be6a-290657ef88da" (UID: "3ed46963-d5f2-4d87-be6a-290657ef88da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:28.831175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.831129 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\") pod \"3ed46963-d5f2-4d87-be6a-290657ef88da\" (UID: \"3ed46963-d5f2-4d87-be6a-290657ef88da\") " Nov 01 05:20:28.831605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.831577 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:28.833682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.833657 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx" pod "3ed46963-d5f2-4d87-be6a-290657ef88da" (UID: "3ed46963-d5f2-4d87-be6a-290657ef88da"). InnerVolumeSpecName "kube-api-access-rwclx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:28.836752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.836726 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca" pod "3ed46963-d5f2-4d87-be6a-290657ef88da" (UID: "3ed46963-d5f2-4d87-be6a-290657ef88da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:28.868180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.868149 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 05:20:28.868480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.868453 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 05:20:28.933000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.932969 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwclx\" (UniqueName: \"kubernetes.io/projected/3ed46963-d5f2-4d87-be6a-290657ef88da-kube-api-access-rwclx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:28.933000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:28.932997 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed46963-d5f2-4d87-be6a-290657ef88da-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:29.071700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.071658 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55"] Nov 01 05:20:29.075606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.075582 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9a82e8-s7m55"] Nov 01 05:20:29.515992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.515964 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" Nov 01 05:20:29.516144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.515991 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf" event={"ID":"3ed46963-d5f2-4d87-be6a-290657ef88da","Type":"ContainerDied","Data":"dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2"} Nov 01 05:20:29.516144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.516036 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd81cdd9416af71ed75f8e605e1255241421735874cd3d4a35779a828c0c11f2" Nov 01 05:20:29.626663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.626628 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7"] Nov 01 05:20:29.627231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.627216 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3ed46963-d5f2-4d87-be6a-290657ef88da" containerName="revoke-keys" Nov 01 05:20:29.627321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.627232 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed46963-d5f2-4d87-be6a-290657ef88da" containerName="revoke-keys" Nov 01 05:20:29.627367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.627358 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed46963-d5f2-4d87-be6a-290657ef88da" containerName="revoke-keys" Nov 01 05:20:29.631085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.631064 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.686042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.685035 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:20:29.686042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.685344 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:20:29.688487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.688460 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7"] Nov 01 05:20:29.693908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.693882 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:20:29.694029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.693989 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:20:29.784428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.781567 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.784428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.781641 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.784428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.781756 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vc9p\" (UniqueName: \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.867693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.867549 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 05:20:29.867693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.867614 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 05:20:29.883510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.883477 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.883694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.883517 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.883694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.883550 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vc9p\" (UniqueName: \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.884166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.884143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.886156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.886124 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:29.967105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:29.967058 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vc9p\" (UniqueName: \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\") pod \"maas-api-revoke-sub-cc9385bf-mdvf7\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:30.264355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:30.260480 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:30.670570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:30.670537 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7"] Nov 01 05:20:30.671800 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:30.671768 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd39080f_fc35_42a6_a0f8_ee7baf6c0c13.slice/crio-453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c WatchSource:0}: Error finding container 453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c: Status 404 returned error can't find the container with id 453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c Nov 01 05:20:30.867301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:30.867172 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 05:20:30.867301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:30.867235 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 05:20:30.974317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:30.974217 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31165c24-7a04-4db2-8939-943577098990" path="/var/lib/kubelet/pods/31165c24-7a04-4db2-8939-943577098990/volumes" Nov 01 05:20:31.533961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:31.533926 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" event={"ID":"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13","Type":"ContainerStarted","Data":"9b5f5ff0c70fd983b4880bc43bfc43ce24277598efa1776203451a27179ebe0d"} Nov 01 05:20:31.534130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:31.533983 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" event={"ID":"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13","Type":"ContainerStarted","Data":"453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c"} Nov 01 05:20:31.656215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:31.656162 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" podStartSLOduration=2.6561485830000002 podStartE2EDuration="2.656148583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:31.652576389 +0000 UTC m=+1046.793165310" watchObservedRunningTime="2026-11-01 05:20:31.656148583 +0000 UTC m=+1046.796737526" Nov 01 05:20:32.034364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:32.034331 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:32.035208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:32.035176 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:32.036450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:32.036428 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:20:32.541965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:32.541929 4425 generic.go:356] "Generic (PLEG): container finished" podID="cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" containerID="9b5f5ff0c70fd983b4880bc43bfc43ce24277598efa1776203451a27179ebe0d" exitCode=0 Nov 01 05:20:32.542167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:32.542015 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" event={"ID":"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13","Type":"ContainerDied","Data":"9b5f5ff0c70fd983b4880bc43bfc43ce24277598efa1776203451a27179ebe0d"} Nov 01 05:20:33.845744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.845717 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:33.938554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.938520 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\") pod \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " Nov 01 05:20:33.938755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.938614 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\" (UniqueName: \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\") pod \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " Nov 01 05:20:33.938755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.938656 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\") pod \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\" (UID: \"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13\") " Nov 01 05:20:33.939074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.939051 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca" pod "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" (UID: "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:33.940938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.940904 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca" pod "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" (UID: "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:33.941039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:33.940966 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p" pod "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" (UID: "cd39080f-fc35-42a6-a0f8-ee7baf6c0c13"). InnerVolumeSpecName "kube-api-access-9vc9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:34.040633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.040602 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:34.040633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.040629 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vc9p\" (UniqueName: \"kubernetes.io/projected/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-kube-api-access-9vc9p\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:34.040633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.040639 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:34.563175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.563138 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" Nov 01 05:20:34.563175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.563153 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7" event={"ID":"cd39080f-fc35-42a6-a0f8-ee7baf6c0c13","Type":"ContainerDied","Data":"453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c"} Nov 01 05:20:34.563480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:34.563201 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453c8f3433811a719a7a4d898b22ddff941edbafdac0b1e77bf67849621fc91c" Nov 01 05:20:36.392153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.392120 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk"] Nov 01 05:20:36.392734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.392720 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" containerName="revoke-keys" Nov 01 05:20:36.392794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.392736 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" containerName="revoke-keys" Nov 01 05:20:36.392848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.392839 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" containerName="revoke-keys" Nov 01 05:20:36.396533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.396517 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.453346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.453310 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk"] Nov 01 05:20:36.499932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.499905 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:20:36.499932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.499905 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:20:36.499932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.499906 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:20:36.501221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.501200 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:20:36.573488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.573458 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.573666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.573496 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsr6n\" (UniqueName: \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.573713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.573685 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.675034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.674950 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.675034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.675018 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.675219 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.675050 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsr6n\" (UniqueName: \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.675853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.675773 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.677488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.677467 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:36.737957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:36.737919 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsr6n\" (UniqueName: \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\") pod \"maas-api-revoke-sub-186a5ede-ltlpk\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:37.009552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:37.009457 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:37.391958 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:37.391925 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5af7c26_7ced_416b_9d69_64291a242fb8.slice/crio-d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829 WatchSource:0}: Error finding container d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829: Status 404 returned error can't find the container with id d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829 Nov 01 05:20:37.392128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:37.392112 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk"] Nov 01 05:20:37.584412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:37.584374 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" event={"ID":"b5af7c26-7ced-416b-9d69-64291a242fb8","Type":"ContainerStarted","Data":"d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829"} Nov 01 05:20:38.235896 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:20:38.235862 4425 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5af7c26_7ced_416b_9d69_64291a242fb8.slice/crio-a55b2aea3c2d0466304c60ce6c86291787a732a083a36b4d30bcc11288593132.scope\": RecentStats: unable to find data in memory cache]" Nov 01 05:20:38.593151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:38.593108 4425 generic.go:356] "Generic (PLEG): container finished" podID="b5af7c26-7ced-416b-9d69-64291a242fb8" containerID="a55b2aea3c2d0466304c60ce6c86291787a732a083a36b4d30bcc11288593132" exitCode=0 Nov 01 05:20:38.593562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:38.593195 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" event={"ID":"b5af7c26-7ced-416b-9d69-64291a242fb8","Type":"ContainerDied","Data":"a55b2aea3c2d0466304c60ce6c86291787a732a083a36b4d30bcc11288593132"} Nov 01 05:20:39.755084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.755057 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:39.818789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.818762 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\") pod \"b5af7c26-7ced-416b-9d69-64291a242fb8\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " Nov 01 05:20:39.818992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.818827 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\") pod \"b5af7c26-7ced-416b-9d69-64291a242fb8\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " Nov 01 05:20:39.818992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.818848 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\" (UniqueName: \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\") pod \"b5af7c26-7ced-416b-9d69-64291a242fb8\" (UID: \"b5af7c26-7ced-416b-9d69-64291a242fb8\") " Nov 01 05:20:39.819194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.819171 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca" pod "b5af7c26-7ced-416b-9d69-64291a242fb8" (UID: "b5af7c26-7ced-416b-9d69-64291a242fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:39.821149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.821113 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca" pod "b5af7c26-7ced-416b-9d69-64291a242fb8" (UID: "b5af7c26-7ced-416b-9d69-64291a242fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:39.821323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.821154 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n" pod "b5af7c26-7ced-416b-9d69-64291a242fb8" (UID: "b5af7c26-7ced-416b-9d69-64291a242fb8"). InnerVolumeSpecName "kube-api-access-jsr6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:39.920528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.920454 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:39.920528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.920481 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5af7c26-7ced-416b-9d69-64291a242fb8-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:39.920528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:39.920491 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsr6n\" (UniqueName: \"kubernetes.io/projected/b5af7c26-7ced-416b-9d69-64291a242fb8-kube-api-access-jsr6n\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:40.609864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:40.609824 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" event={"ID":"b5af7c26-7ced-416b-9d69-64291a242fb8","Type":"ContainerDied","Data":"d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829"} Nov 01 05:20:40.609864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:40.609864 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk" Nov 01 05:20:40.610087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:40.609886 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ad2df83b94d8f2db28a200f7306a753ddda6542b8520268c676cad6e176829" Nov 01 05:20:53.188212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.188173 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh"] Nov 01 05:20:53.188861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.188820 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b5af7c26-7ced-416b-9d69-64291a242fb8" containerName="revoke-keys" Nov 01 05:20:53.188861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.188834 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5af7c26-7ced-416b-9d69-64291a242fb8" containerName="revoke-keys" Nov 01 05:20:53.188997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.188920 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5af7c26-7ced-416b-9d69-64291a242fb8" containerName="revoke-keys" Nov 01 05:20:53.192567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.192544 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.247069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.247034 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:20:53.249858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.249832 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh"] Nov 01 05:20:53.295436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.295411 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:20:53.295621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.295457 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:20:53.295621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.295469 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:20:53.309020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.308993 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.309156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.309069 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gp5b\" (UniqueName: \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.309310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.309287 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.410675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.410637 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gp5b\" (UniqueName: \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.410861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.410838 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.410920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.410898 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.411561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.411536 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.413493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.413464 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.473116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.473041 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gp5b\" (UniqueName: \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\") pod \"maas-api-revoke-sub-9e4adcec-tv8qh\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.506704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.506671 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:53.886999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:53.886973 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh"] Nov 01 05:20:53.887143 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:20:53.887062 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2496da3e_da85_4f81_abd1_662ac351d46b.slice/crio-ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff WatchSource:0}: Error finding container ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff: Status 404 returned error can't find the container with id ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff Nov 01 05:20:54.709112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:54.709082 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" event={"ID":"2496da3e-da85-4f81-abd1-662ac351d46b","Type":"ContainerStarted","Data":"45ab2f926f90c52f6645e5b18438173fe3383a9af1c1118eabff8babaf41b261"} Nov 01 05:20:54.709535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:54.709145 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" event={"ID":"2496da3e-da85-4f81-abd1-662ac351d46b","Type":"ContainerStarted","Data":"ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff"} Nov 01 05:20:54.829286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:54.829194 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" podStartSLOduration=1.8291803340000001 podStartE2EDuration="1.829180334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:54.826492568 +0000 UTC m=+1069.967081516" watchObservedRunningTime="2026-11-01 05:20:54.829180334 +0000 UTC m=+1069.969769252" Nov 01 05:20:55.717718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:55.717685 4425 generic.go:356] "Generic (PLEG): container finished" podID="2496da3e-da85-4f81-abd1-662ac351d46b" containerID="45ab2f926f90c52f6645e5b18438173fe3383a9af1c1118eabff8babaf41b261" exitCode=0 Nov 01 05:20:55.718120 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:55.717780 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" event={"ID":"2496da3e-da85-4f81-abd1-662ac351d46b","Type":"ContainerDied","Data":"45ab2f926f90c52f6645e5b18438173fe3383a9af1c1118eabff8babaf41b261"} Nov 01 05:20:56.882006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:56.881971 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:56.978518 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:20:56.978442 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:20:57.055925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.055890 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\") pod \"2496da3e-da85-4f81-abd1-662ac351d46b\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " Nov 01 05:20:57.056126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.055947 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\" (UniqueName: \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\") pod \"2496da3e-da85-4f81-abd1-662ac351d46b\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " Nov 01 05:20:57.056126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.055987 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\") pod \"2496da3e-da85-4f81-abd1-662ac351d46b\" (UID: \"2496da3e-da85-4f81-abd1-662ac351d46b\") " Nov 01 05:20:57.056367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.056342 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca" pod "2496da3e-da85-4f81-abd1-662ac351d46b" (UID: "2496da3e-da85-4f81-abd1-662ac351d46b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:20:57.056541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.056515 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:57.058346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.058324 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca" pod "2496da3e-da85-4f81-abd1-662ac351d46b" (UID: "2496da3e-da85-4f81-abd1-662ac351d46b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:20:57.058435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.058418 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b" pod "2496da3e-da85-4f81-abd1-662ac351d46b" (UID: "2496da3e-da85-4f81-abd1-662ac351d46b"). InnerVolumeSpecName "kube-api-access-8gp5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:20:57.157199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.157167 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gp5b\" (UniqueName: \"kubernetes.io/projected/2496da3e-da85-4f81-abd1-662ac351d46b-kube-api-access-8gp5b\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:57.157199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.157195 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2496da3e-da85-4f81-abd1-662ac351d46b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:20:57.733942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.733903 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" event={"ID":"2496da3e-da85-4f81-abd1-662ac351d46b","Type":"ContainerDied","Data":"ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff"} Nov 01 05:20:57.733942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.733935 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh" Nov 01 05:20:57.734164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:20:57.733956 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad86d94f1498690572f8d2ba11d902ba9a5a5df3bea152a31d056b016760f3ff" Nov 01 05:21:00.956420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:00.956397 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:00.956958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:00.956945 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2496da3e-da85-4f81-abd1-662ac351d46b" containerName="revoke-keys" Nov 01 05:21:00.957003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:00.956960 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2496da3e-da85-4f81-abd1-662ac351d46b" containerName="revoke-keys" Nov 01 05:21:00.957065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:00.957056 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2496da3e-da85-4f81-abd1-662ac351d46b" containerName="revoke-keys" Nov 01 05:21:00.960907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:00.960891 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.014883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.014851 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-17306a2b8c16\"" Nov 01 05:21:01.014883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.014873 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\"" Nov 01 05:21:01.018214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.018185 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:01.065079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.065049 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\"" Nov 01 05:21:01.065079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.065051 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\"" Nov 01 05:21:01.065079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.065051 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\"" Nov 01 05:21:01.065360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.065050 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\"" Nov 01 05:21:01.065360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.065096 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\"" Nov 01 05:21:01.103173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.103138 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.103360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.103278 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.103415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.103398 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.103449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.103431 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz22\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.205029 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.205121 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.205193 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.205224 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz22\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.205913 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.205991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.206004 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.207914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.207892 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.267659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.267625 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz22\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.274564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.274540 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.655054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.655020 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:01.655647 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:21:01.655616 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fc07bf_d7f7_49c8_b486_36b01da96ca9.slice/crio-1b7885a9002eb7776f1de542c51355831eac01a698cb42b18fbee734763c06f3 WatchSource:0}: Error finding container 1b7885a9002eb7776f1de542c51355831eac01a698cb42b18fbee734763c06f3: Status 404 returned error can't find the container with id 1b7885a9002eb7776f1de542c51355831eac01a698cb42b18fbee734763c06f3 Nov 01 05:21:01.763115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.763073 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" event={"ID":"66fc07bf-d7f7-49c8-b486-36b01da96ca9","Type":"ContainerStarted","Data":"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684"} Nov 01 05:21:01.763373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.763135 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:01.763373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.763159 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" event={"ID":"66fc07bf-d7f7-49c8-b486-36b01da96ca9","Type":"ContainerStarted","Data":"1b7885a9002eb7776f1de542c51355831eac01a698cb42b18fbee734763c06f3"} Nov 01 05:21:01.885288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:01.885206 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" podStartSLOduration=1.8851761360000001 podStartE2EDuration="1.885176136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:01.881766773 +0000 UTC m=+1077.022355695" watchObservedRunningTime="2026-11-01 05:21:01.885176136 +0000 UTC m=+1077.025765054" Nov 01 05:21:10.077279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.076299 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp"] Nov 01 05:21:10.095282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.091874 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.111151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.111119 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp"] Nov 01 05:21:10.156522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.156491 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:21:10.156762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.156528 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:21:10.156762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.156528 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:21:10.156762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.156543 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:21:10.207312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.207278 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.207490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.207445 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvvd\" (UniqueName: \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.207490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.207478 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.309036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.308997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvvd\" (UniqueName: \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.309246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.309048 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.309246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.309171 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.309825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.309797 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.311710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.311690 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.372083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.372016 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvvd\" (UniqueName: \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\") pod \"maas-api-revoke-sub-45c04b93-9cqfp\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.416040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.416003 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:10.834460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:10.834428 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp"] Nov 01 05:21:10.839605 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:21:10.839577 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6638d2_6339_47ab_8f5e_361e51ae826b.slice/crio-962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b WatchSource:0}: Error finding container 962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b: Status 404 returned error can't find the container with id 962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b Nov 01 05:21:11.838491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:11.838458 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" event={"ID":"3f6638d2-6339-47ab-8f5e-361e51ae826b","Type":"ContainerStarted","Data":"69cb6a1453a453ad02b33cad4c4f08064c58bbc3399cfd04162e176c7a75f6fa"} Nov 01 05:21:11.838974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:11.838536 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" event={"ID":"3f6638d2-6339-47ab-8f5e-361e51ae826b","Type":"ContainerStarted","Data":"962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b"} Nov 01 05:21:11.971322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:11.967116 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" podStartSLOduration=1.967095665 podStartE2EDuration="1.967095665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:11.964460352 +0000 UTC m=+1087.105049283" watchObservedRunningTime="2026-11-01 05:21:11.967095665 +0000 UTC m=+1087.107684584" Nov 01 05:21:12.775083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:12.775049 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:12.873565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:12.873531 4425 generic.go:356] "Generic (PLEG): container finished" podID="3f6638d2-6339-47ab-8f5e-361e51ae826b" containerID="69cb6a1453a453ad02b33cad4c4f08064c58bbc3399cfd04162e176c7a75f6fa" exitCode=0 Nov 01 05:21:12.874058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:12.873579 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" event={"ID":"3f6638d2-6339-47ab-8f5e-361e51ae826b","Type":"ContainerDied","Data":"69cb6a1453a453ad02b33cad4c4f08064c58bbc3399cfd04162e176c7a75f6fa"} Nov 01 05:21:14.066609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.066573 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:14.191409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.191328 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\" (UniqueName: \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\") pod \"3f6638d2-6339-47ab-8f5e-361e51ae826b\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " Nov 01 05:21:14.191559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.191424 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\") pod \"3f6638d2-6339-47ab-8f5e-361e51ae826b\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " Nov 01 05:21:14.191559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.191476 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\") pod \"3f6638d2-6339-47ab-8f5e-361e51ae826b\" (UID: \"3f6638d2-6339-47ab-8f5e-361e51ae826b\") " Nov 01 05:21:14.191819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.191788 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca" pod "3f6638d2-6339-47ab-8f5e-361e51ae826b" (UID: "3f6638d2-6339-47ab-8f5e-361e51ae826b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:14.191993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.191976 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:14.193747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.193718 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca" pod "3f6638d2-6339-47ab-8f5e-361e51ae826b" (UID: "3f6638d2-6339-47ab-8f5e-361e51ae826b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:21:14.193934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.193914 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd" pod "3f6638d2-6339-47ab-8f5e-361e51ae826b" (UID: "3f6638d2-6339-47ab-8f5e-361e51ae826b"). InnerVolumeSpecName "kube-api-access-shvvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:14.293672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.293636 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6638d2-6339-47ab-8f5e-361e51ae826b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:14.293672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.293670 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shvvd\" (UniqueName: \"kubernetes.io/projected/3f6638d2-6339-47ab-8f5e-361e51ae826b-kube-api-access-shvvd\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:14.890484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.890454 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" Nov 01 05:21:14.890484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.890484 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp" event={"ID":"3f6638d2-6339-47ab-8f5e-361e51ae826b","Type":"ContainerDied","Data":"962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b"} Nov 01 05:21:14.890712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:14.890542 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962c59dd71a2284acfe323dbb2f5d2fb550c3f71b9af10d4ce5d942459c9ab7b" Nov 01 05:21:21.333487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.333452 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:21.333986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.333522 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:21.942283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.942219 4425 generic.go:356] "Generic (PLEG): container finished" podID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerID="78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1" exitCode=0 Nov 01 05:21:21.942488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.942300 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerDied","Data":"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1"} Nov 01 05:21:21.942804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.942786 4425 scope.go:122] "RemoveContainer" containerID="78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1" Nov 01 05:21:21.944226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.944199 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:21.944324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.944287 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:21.944368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:21.944325 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:22.975690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:22.975654 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerStarted","Data":"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82"} Nov 01 05:21:23.673008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.672970 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:21:23.673228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.673210 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:23.673306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.673286 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:23.967336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.967225 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 05:21:23.968092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.968054 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3f6638d2-6339-47ab-8f5e-361e51ae826b" containerName="revoke-keys" Nov 01 05:21:23.968092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.968087 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6638d2-6339-47ab-8f5e-361e51ae826b" containerName="revoke-keys" Nov 01 05:21:23.968356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.968330 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6638d2-6339-47ab-8f5e-361e51ae826b" containerName="revoke-keys" Nov 01 05:21:23.972403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:23.972381 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.085090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.085050 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 05:21:24.111327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111278 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111314 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111424 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111479 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111588 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111746 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111803 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.111959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111921 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.112030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.111991 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcdq\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.213736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213702 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.213931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213788 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.213931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213833 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.213931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213881 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.213931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213919 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcdq\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.213995 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214022 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214215 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214286 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214316 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214358 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214521 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214545 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.214977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.214953 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.216355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.216330 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.216664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.216646 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.277175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.277086 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcdq\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.277175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.277155 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-j8mfj\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.287089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.287060 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:24.673184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.673153 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:24.673413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.673231 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:24.787387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.770146 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 05:21:24.799756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.795488 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:24.799756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.795635 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:24.799756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:24.795695 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:21:25.004209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.004122 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" event={"ID":"e8a31114-3554-4a28-b394-8ca65cd2684c","Type":"ContainerStarted","Data":"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4"} Nov 01 05:21:25.004209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.004202 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" event={"ID":"e8a31114-3554-4a28-b394-8ca65cd2684c","Type":"ContainerStarted","Data":"eb88a783c7326d84ac08fc396cc38977521a97169db22c392b5696e32df8033c"} Nov 01 05:21:25.138902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.138839 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podStartSLOduration=2.138818934 podStartE2EDuration="2.138818934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:25.133281624 +0000 UTC m=+1100.273870545" watchObservedRunningTime="2026-11-01 05:21:25.138818934 +0000 UTC m=+1100.279407854" Nov 01 05:21:25.287862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.287782 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:25.289058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.289026 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:25.289192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.289124 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:25.628929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.628898 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z"] Nov 01 05:21:25.633577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.633553 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.674666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.673413 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:25.674666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.673473 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:25.704119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.704097 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:21:25.707561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.704381 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:21:25.707561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.704512 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:21:25.707561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.704592 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:21:25.711304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.709001 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z"] Nov 01 05:21:25.733656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.733629 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.733790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.733747 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslsf\" (UniqueName: \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.733790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.733773 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.835248 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.835212 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslsf\" (UniqueName: \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.835434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.835276 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.835434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.835337 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.836059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.836032 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.840723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.840704 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:25.901750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:25.901663 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslsf\" (UniqueName: \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\") pod \"maas-api-revoke-sub-65604218-rb48z\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:26.004959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.004865 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:26.297302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.296456 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:26.297302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.296524 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:26.587376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.587318 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z"] Nov 01 05:21:26.595285 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:21:26.589759 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae4fea1_3707_4fe3_bc87_753dd4f219b4.slice/crio-90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581 WatchSource:0}: Error finding container 90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581: Status 404 returned error can't find the container with id 90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581 Nov 01 05:21:26.687279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.673731 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:26.687279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:26.673794 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:27.000206 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:21:27.000171 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:21:27.027600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:27.027559 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" event={"ID":"aae4fea1-3707-4fe3-bc87-753dd4f219b4","Type":"ContainerStarted","Data":"90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581"} Nov 01 05:21:27.288342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:27.287830 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:27.288342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:27.287891 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:27.681282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:27.675797 4425 patch_prober.go:29] interesting pod/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 05:21:27.681282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:27.675875 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 05:21:28.038492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.038397 4425 generic.go:356] "Generic (PLEG): container finished" podID="aae4fea1-3707-4fe3-bc87-753dd4f219b4" containerID="03da23c2f7647c304214749e8c121fa22eb5c78d92f5acffe2d6c5c120305fa5" exitCode=0 Nov 01 05:21:28.038646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.038486 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" event={"ID":"aae4fea1-3707-4fe3-bc87-753dd4f219b4","Type":"ContainerDied","Data":"03da23c2f7647c304214749e8c121fa22eb5c78d92f5acffe2d6c5c120305fa5"} Nov 01 05:21:28.287923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.287881 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:28.288108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.287939 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:28.678852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.678820 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:21:28.679302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.679281 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:21:28.680147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:28.680132 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:21:29.287229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.287202 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:29.287762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.287737 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:29.287852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.287789 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:29.398747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.398671 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\") pod \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " Nov 01 05:21:29.398892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.398874 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\" (UniqueName: \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\") pod \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " Nov 01 05:21:29.398934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.398924 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\") pod \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\" (UID: \"aae4fea1-3707-4fe3-bc87-753dd4f219b4\") " Nov 01 05:21:29.399128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.399099 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca" pod "aae4fea1-3707-4fe3-bc87-753dd4f219b4" (UID: "aae4fea1-3707-4fe3-bc87-753dd4f219b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:29.399341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.399315 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:29.401141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.401113 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf" pod "aae4fea1-3707-4fe3-bc87-753dd4f219b4" (UID: "aae4fea1-3707-4fe3-bc87-753dd4f219b4"). InnerVolumeSpecName "kube-api-access-kslsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:29.401287 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.401201 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca" pod "aae4fea1-3707-4fe3-bc87-753dd4f219b4" (UID: "aae4fea1-3707-4fe3-bc87-753dd4f219b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:21:29.501100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.501060 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kslsf\" (UniqueName: \"kubernetes.io/projected/aae4fea1-3707-4fe3-bc87-753dd4f219b4-kube-api-access-kslsf\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:29.501100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:29.501101 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4fea1-3707-4fe3-bc87-753dd4f219b4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:30.095040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:30.095011 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" Nov 01 05:21:30.095194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:30.095049 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z" event={"ID":"aae4fea1-3707-4fe3-bc87-753dd4f219b4","Type":"ContainerDied","Data":"90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581"} Nov 01 05:21:30.095194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:30.095104 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90523b6c1ca1fe0ab40381c5c6c93ecf8f7fd9efd497d7ce30b82a0b0c2d0581" Nov 01 05:21:30.287924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:30.287893 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:30.288428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:30.287950 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:31.287801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:31.287763 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 05:21:31.287978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:31.287840 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 05:21:32.435834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.435796 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:32.436331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.436196 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:32.437089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.437071 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 05:21:32.681908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.681877 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:32.975409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.975374 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr"] Nov 01 05:21:32.975984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.975964 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aae4fea1-3707-4fe3-bc87-753dd4f219b4" containerName="revoke-keys" Nov 01 05:21:32.976086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.975986 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae4fea1-3707-4fe3-bc87-753dd4f219b4" containerName="revoke-keys" Nov 01 05:21:32.976203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.976189 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae4fea1-3707-4fe3-bc87-753dd4f219b4" containerName="revoke-keys" Nov 01 05:21:32.979945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:32.979924 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.031890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.031851 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr"] Nov 01 05:21:33.033638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.033609 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:21:33.034130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.034106 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:21:33.034277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.034169 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:21:33.034277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.034178 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:21:33.048924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.048893 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.049058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.048930 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwg8\" (UniqueName: \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.049184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.049165 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.150592 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.150552 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.150809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.150606 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwg8\" (UniqueName: \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.150809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.150696 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.151448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.151412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.153512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.153483 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.213534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.213501 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwg8\" (UniqueName: \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\") pod \"maas-api-revoke-sub-0e695195-2qqpr\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.293822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.293735 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:33.687172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:33.687133 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr"] Nov 01 05:21:33.689017 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:21:33.688993 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb8dd2_8b82_46b4_8a54_065b1a5a251a.slice/crio-fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6 WatchSource:0}: Error finding container fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6: Status 404 returned error can't find the container with id fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6 Nov 01 05:21:34.132035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:34.131992 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" event={"ID":"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a","Type":"ContainerStarted","Data":"a5004c6178f0edc66774082b4d5aa90edc1bedba32c296052c2369a3d112d5d0"} Nov 01 05:21:34.132228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:34.132053 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" event={"ID":"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a","Type":"ContainerStarted","Data":"fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6"} Nov 01 05:21:34.253792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:34.253684 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" podStartSLOduration=2.253668818 podStartE2EDuration="2.253668818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:34.249339841 +0000 UTC m=+1109.389928761" watchObservedRunningTime="2026-11-01 05:21:34.253668818 +0000 UTC m=+1109.394257737" Nov 01 05:21:35.140580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:35.140545 4425 generic.go:356] "Generic (PLEG): container finished" podID="b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" containerID="a5004c6178f0edc66774082b4d5aa90edc1bedba32c296052c2369a3d112d5d0" exitCode=0 Nov 01 05:21:35.140973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:35.140629 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" event={"ID":"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a","Type":"ContainerDied","Data":"a5004c6178f0edc66774082b4d5aa90edc1bedba32c296052c2369a3d112d5d0"} Nov 01 05:21:36.336509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.336481 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:36.396689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.396646 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\" (UniqueName: \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\") pod \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " Nov 01 05:21:36.396842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.396775 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\") pod \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " Nov 01 05:21:36.396842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.396814 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\") pod \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\" (UID: \"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a\") " Nov 01 05:21:36.397162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.397137 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca" pod "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" (UID: "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:36.399078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.399045 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca" pod "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" (UID: "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:21:36.399078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.399065 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8" pod "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" (UID: "b6eb8dd2-8b82-46b4-8a54-065b1a5a251a"). InnerVolumeSpecName "kube-api-access-tzwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:36.498816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.498777 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:36.498816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.498811 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:36.498816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:36.498820 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwg8\" (UniqueName: \"kubernetes.io/projected/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a-kube-api-access-tzwg8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:37.159911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:37.159876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" event={"ID":"b6eb8dd2-8b82-46b4-8a54-065b1a5a251a","Type":"ContainerDied","Data":"fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6"} Nov 01 05:21:37.160081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:37.159921 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr" Nov 01 05:21:37.160081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:37.159931 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde40b73117face620ca626840482d3b74effa0d0ca0ad019c4e8e18a0e7a7c6" Nov 01 05:21:37.683096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:37.683024 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" podUID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" containerName="payload-processing" containerID="cri-o://eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684" gracePeriod=25 Nov 01 05:21:37.949542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:37.949482 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:38.019687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.019656 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\") pod \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " Nov 01 05:21:38.019827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.019773 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\") pod \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " Nov 01 05:21:38.019827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.019798 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\") pod \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " Nov 01 05:21:38.019950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.019837 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\") pod \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\" (UID: \"66fc07bf-d7f7-49c8-b486-36b01da96ca9\") " Nov 01 05:21:38.020202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.020174 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume" pod "66fc07bf-d7f7-49c8-b486-36b01da96ca9" (UID: "66fc07bf-d7f7-49c8-b486-36b01da96ca9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:38.020348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.020223 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume" pod "66fc07bf-d7f7-49c8-b486-36b01da96ca9" (UID: "66fc07bf-d7f7-49c8-b486-36b01da96ca9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:38.020715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.020697 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:38.020759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.020722 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66fc07bf-d7f7-49c8-b486-36b01da96ca9-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:38.022291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.022265 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials" pod "66fc07bf-d7f7-49c8-b486-36b01da96ca9" (UID: "66fc07bf-d7f7-49c8-b486-36b01da96ca9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:38.022383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.022300 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22" pod "66fc07bf-d7f7-49c8-b486-36b01da96ca9" (UID: "66fc07bf-d7f7-49c8-b486-36b01da96ca9"). InnerVolumeSpecName "kube-api-access-gmz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:38.122365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.122330 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:38.122365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.122359 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmz22\" (UniqueName: \"kubernetes.io/projected/66fc07bf-d7f7-49c8-b486-36b01da96ca9-kube-api-access-gmz22\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:38.168394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.168362 4425 generic.go:356] "Generic (PLEG): container finished" podID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" containerID="eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684" exitCode=0 Nov 01 05:21:38.168534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.168409 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" Nov 01 05:21:38.168534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.168448 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" event={"ID":"66fc07bf-d7f7-49c8-b486-36b01da96ca9","Type":"ContainerDied","Data":"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684"} Nov 01 05:21:38.168534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.168502 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v" event={"ID":"66fc07bf-d7f7-49c8-b486-36b01da96ca9","Type":"ContainerDied","Data":"1b7885a9002eb7776f1de542c51355831eac01a698cb42b18fbee734763c06f3"} Nov 01 05:21:38.168534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.168524 4425 scope.go:122] "RemoveContainer" containerID="eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684" Nov 01 05:21:38.183364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.183343 4425 scope.go:122] "RemoveContainer" containerID="eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684" Nov 01 05:21:38.183699 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:21:38.183677 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684\": container with ID starting with eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684 not found: ID does not exist" containerID="eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684" Nov 01 05:21:38.183765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.183712 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684"} err="failed to get container status \"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684\": rpc error: code = NotFound desc = could not find container \"eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684\": container with ID starting with eada31674c7fe4111d9eac775f99613a87f91d637941b9237bf78d120fe9b684 not found: ID does not exist" Nov 01 05:21:38.339062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.339032 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:38.342773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.342747 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7w7k2v"] Nov 01 05:21:38.973795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:38.973759 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" path="/var/lib/kubelet/pods/66fc07bf-d7f7-49c8-b486-36b01da96ca9/volumes" Nov 01 05:21:41.621074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:41.621011 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:21:46.622230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:46.622130 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" podUID="64641330-77d1-44af-b67d-c140db584523" containerName="payload-processing" containerID="cri-o://856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7" gracePeriod=25 Nov 01 05:21:46.941765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:46.941726 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:21:47.065599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.061975 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\") pod \"64641330-77d1-44af-b67d-c140db584523\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " Nov 01 05:21:47.065599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.062075 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\" (UniqueName: \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\") pod \"64641330-77d1-44af-b67d-c140db584523\" (UID: \"64641330-77d1-44af-b67d-c140db584523\") " Nov 01 05:21:47.065599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.063884 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume" pod "64641330-77d1-44af-b67d-c140db584523" (UID: "64641330-77d1-44af-b67d-c140db584523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:21:47.076438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.075756 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw" pod "64641330-77d1-44af-b67d-c140db584523" (UID: "64641330-77d1-44af-b67d-c140db584523"). InnerVolumeSpecName "kube-api-access-qwxtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:21:47.164693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.164578 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64641330-77d1-44af-b67d-c140db584523-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:47.164693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.164626 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxtw\" (UniqueName: \"kubernetes.io/projected/64641330-77d1-44af-b67d-c140db584523-kube-api-access-qwxtw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:21:47.268142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.268098 4425 generic.go:356] "Generic (PLEG): container finished" podID="64641330-77d1-44af-b67d-c140db584523" containerID="856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7" exitCode=0 Nov 01 05:21:47.268356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.268150 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" Nov 01 05:21:47.268356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.268185 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" event={"ID":"64641330-77d1-44af-b67d-c140db584523","Type":"ContainerDied","Data":"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7"} Nov 01 05:21:47.268356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.268298 4425 scope.go:122] "RemoveContainer" containerID="856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7" Nov 01 05:21:47.268568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.268451 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h" event={"ID":"64641330-77d1-44af-b67d-c140db584523","Type":"ContainerDied","Data":"97c77b5e16d6f79a9e4b9c229f7c459ff14069157d3fa2af2b57fd1fae615cfe"} Nov 01 05:21:47.283517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.283486 4425 scope.go:122] "RemoveContainer" containerID="856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7" Nov 01 05:21:47.284012 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:21:47.283927 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7\": container with ID starting with 856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7 not found: ID does not exist" containerID="856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7" Nov 01 05:21:47.284012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.283966 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7"} err="failed to get container status \"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7\": rpc error: code = NotFound desc = could not find container \"856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7\": container with ID starting with 856dc035acfb90c2b8bdf818d5cd483b609db3f05e3954e043c617cd6bb6a4c7 not found: ID does not exist" Nov 01 05:21:47.448333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.448179 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:21:47.463079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:47.457846 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-768dff758f-r9m4h"] Nov 01 05:21:48.973919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:48.973884 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64641330-77d1-44af-b67d-c140db584523" path="/var/lib/kubelet/pods/64641330-77d1-44af-b67d-c140db584523/volumes" Nov 01 05:21:50.944036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.943993 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:21:50.944592 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944576 4425 cpu_manager.go:414] "containerMap: removing container" podUID="64641330-77d1-44af-b67d-c140db584523" containerName="payload-processing" Nov 01 05:21:50.944592 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944592 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64641330-77d1-44af-b67d-c140db584523" containerName="payload-processing" Nov 01 05:21:50.944696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944601 4425 cpu_manager.go:414] "containerMap: removing container" podUID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" containerName="payload-processing" Nov 01 05:21:50.944696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944606 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" containerName="payload-processing" Nov 01 05:21:50.944696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944622 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" containerName="revoke-keys" Nov 01 05:21:50.944696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944627 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" containerName="revoke-keys" Nov 01 05:21:50.944864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944727 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" containerName="revoke-keys" Nov 01 05:21:50.944864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944741 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="64641330-77d1-44af-b67d-c140db584523" containerName="payload-processing" Nov 01 05:21:50.944864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.944746 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fc07bf-d7f7-49c8-b486-36b01da96ca9" containerName="payload-processing" Nov 01 05:21:50.949911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:50.949886 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.006905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.006873 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5451cb-dockercfg-l5swb\"" Nov 01 05:21:51.007705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.007661 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5451cb\"" Nov 01 05:21:51.008075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.008055 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:21:51.078984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.078938 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp25b\" (UniqueName: \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.079181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.079162 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.180648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.180612 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp25b\" (UniqueName: \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.180860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.180752 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.181506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.181482 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.243173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.243092 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp25b\" (UniqueName: \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\") pod \"payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.269050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.269005 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:51.649950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:51.649916 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:21:51.650149 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:21:51.650131 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1e42d2_6a87_434c_96d9_c32438e3cdfa.slice/crio-b64906d4345958cde16bb4f92974be98d6fe87c40c1f15462e6dbe5103e157ec WatchSource:0}: Error finding container b64906d4345958cde16bb4f92974be98d6fe87c40c1f15462e6dbe5103e157ec: Status 404 returned error can't find the container with id b64906d4345958cde16bb4f92974be98d6fe87c40c1f15462e6dbe5103e157ec Nov 01 05:21:52.388814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:52.388781 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" event={"ID":"aa1e42d2-6a87-434c-96d9-c32438e3cdfa","Type":"ContainerStarted","Data":"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3"} Nov 01 05:21:52.389238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:52.388845 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" event={"ID":"aa1e42d2-6a87-434c-96d9-c32438e3cdfa","Type":"ContainerStarted","Data":"b64906d4345958cde16bb4f92974be98d6fe87c40c1f15462e6dbe5103e157ec"} Nov 01 05:21:52.389238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:52.388873 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:21:52.512306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:21:52.512234 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" podStartSLOduration=2.512216141 podStartE2EDuration="2.512216141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:52.507465844 +0000 UTC m=+1127.648054765" watchObservedRunningTime="2026-11-01 05:21:52.512216141 +0000 UTC m=+1127.652805086" Nov 01 05:21:56.948232 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:21:56.948190 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:22:01.200126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.200092 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz"] Nov 01 05:22:01.204728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.204683 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.258982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.258951 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:01.259242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.258980 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:01.259242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.259075 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:01.259242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.259156 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:01.263645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.263616 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz"] Nov 01 05:22:01.317217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.317177 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvbv\" (UniqueName: \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.317411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.317369 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.317451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.317430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.419052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.419005 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.419324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.419112 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvbv\" (UniqueName: \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.419324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.419195 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.419863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.419843 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.421636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.421609 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.482155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.482069 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvbv\" (UniqueName: \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\") pod \"maas-api-revoke-sub-6e67ede9-bvxnz\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.523760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.523725 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:01.904765 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:01.904734 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17643fb3_e251_4a88_b7aa_06a591d4814a.slice/crio-6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904 WatchSource:0}: Error finding container 6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904: Status 404 returned error can't find the container with id 6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904 Nov 01 05:22:01.904994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:01.904972 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz"] Nov 01 05:22:02.463564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:02.463472 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" event={"ID":"17643fb3-e251-4a88-b7aa-06a591d4814a","Type":"ContainerStarted","Data":"3778870fe14267d2e87c546b0e4211fb1a722915bdec12c7d85de5cdbb38dd58"} Nov 01 05:22:02.463564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:02.463538 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" event={"ID":"17643fb3-e251-4a88-b7aa-06a591d4814a","Type":"ContainerStarted","Data":"6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904"} Nov 01 05:22:02.586119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:02.586058 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" podStartSLOduration=1.586041556 podStartE2EDuration="1.586041556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:02.581894491 +0000 UTC m=+1137.722483430" watchObservedRunningTime="2026-11-01 05:22:02.586041556 +0000 UTC m=+1137.726630508" Nov 01 05:22:03.399614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:03.399586 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:22:03.472402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:03.472366 4425 generic.go:356] "Generic (PLEG): container finished" podID="17643fb3-e251-4a88-b7aa-06a591d4814a" containerID="3778870fe14267d2e87c546b0e4211fb1a722915bdec12c7d85de5cdbb38dd58" exitCode=0 Nov 01 05:22:03.472802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:03.472460 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" event={"ID":"17643fb3-e251-4a88-b7aa-06a591d4814a","Type":"ContainerDied","Data":"3778870fe14267d2e87c546b0e4211fb1a722915bdec12c7d85de5cdbb38dd58"} Nov 01 05:22:04.644351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.644322 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:04.761979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.761882 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\") pod \"17643fb3-e251-4a88-b7aa-06a591d4814a\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " Nov 01 05:22:04.762162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.762014 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\" (UniqueName: \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\") pod \"17643fb3-e251-4a88-b7aa-06a591d4814a\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " Nov 01 05:22:04.762162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.762146 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\") pod \"17643fb3-e251-4a88-b7aa-06a591d4814a\" (UID: \"17643fb3-e251-4a88-b7aa-06a591d4814a\") " Nov 01 05:22:04.763018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.762974 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca" pod "17643fb3-e251-4a88-b7aa-06a591d4814a" (UID: "17643fb3-e251-4a88-b7aa-06a591d4814a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:04.763600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.763564 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:04.764922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.764880 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca" pod "17643fb3-e251-4a88-b7aa-06a591d4814a" (UID: "17643fb3-e251-4a88-b7aa-06a591d4814a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:04.767676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.767650 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv" pod "17643fb3-e251-4a88-b7aa-06a591d4814a" (UID: "17643fb3-e251-4a88-b7aa-06a591d4814a"). InnerVolumeSpecName "kube-api-access-fzvbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:04.864934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.864907 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17643fb3-e251-4a88-b7aa-06a591d4814a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:04.864934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:04.864934 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvbv\" (UniqueName: \"kubernetes.io/projected/17643fb3-e251-4a88-b7aa-06a591d4814a-kube-api-access-fzvbv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:05.106918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:05.106890 4425 scope.go:122] "RemoveContainer" containerID="5df4e93c62052f9d0853a5f758d553c603ff6e01c8fc1fb1975b81591b003b8d" Nov 01 05:22:05.488107 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:05.488027 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" Nov 01 05:22:05.488274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:05.488027 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz" event={"ID":"17643fb3-e251-4a88-b7aa-06a591d4814a","Type":"ContainerDied","Data":"6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904"} Nov 01 05:22:05.488274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:05.488170 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ddc428822af610a81025b751aee353ef2d365dda29eefbaded41deb51650904" Nov 01 05:22:07.743949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.743912 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr"] Nov 01 05:22:07.744498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.744484 4425 cpu_manager.go:414] "containerMap: removing container" podUID="17643fb3-e251-4a88-b7aa-06a591d4814a" containerName="revoke-keys" Nov 01 05:22:07.744498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.744498 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17643fb3-e251-4a88-b7aa-06a591d4814a" containerName="revoke-keys" Nov 01 05:22:07.744628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.744619 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="17643fb3-e251-4a88-b7aa-06a591d4814a" containerName="revoke-keys" Nov 01 05:22:07.749738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.749719 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.800757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.800722 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktcgr\" (UniqueName: \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.800948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.800796 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.801155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.801133 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.806022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.806000 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr"] Nov 01 05:22:07.852496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.852467 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:07.852662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.852619 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:07.854468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.854441 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:07.854656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.854636 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:07.902629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.902597 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.902806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.902667 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktcgr\" (UniqueName: \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.902806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.902710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.903389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.903370 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.905097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.905073 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:07.966008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:07.965976 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktcgr\" (UniqueName: \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\") pod \"maas-api-revoke-sub-7e2b2e85-2vqkr\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:08.063395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:08.063374 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:08.442161 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:08.442133 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ecb737_5d04_414b_af48_13250ec0f214.slice/crio-c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d WatchSource:0}: Error finding container c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d: Status 404 returned error can't find the container with id c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d Nov 01 05:22:08.442654 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:08.442633 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr"] Nov 01 05:22:08.511828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:08.511793 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" event={"ID":"68ecb737-5d04-414b-af48-13250ec0f214","Type":"ContainerStarted","Data":"c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d"} Nov 01 05:22:09.520111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:09.520071 4425 generic.go:356] "Generic (PLEG): container finished" podID="68ecb737-5d04-414b-af48-13250ec0f214" containerID="d246b463fb123d42e9d17ab5c2d7f738704f70574d145987600868ce1412102e" exitCode=0 Nov 01 05:22:09.520543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:09.520135 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" event={"ID":"68ecb737-5d04-414b-af48-13250ec0f214","Type":"ContainerDied","Data":"d246b463fb123d42e9d17ab5c2d7f738704f70574d145987600868ce1412102e"} Nov 01 05:22:10.366587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.366555 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:22:10.370792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.370775 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.417677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.417645 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:22:10.421904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.421880 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.433742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.433714 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.433875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.433767 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.433875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.433843 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.433994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.433904 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdxq\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.433994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.433933 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.434101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.434084 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.434138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.434114 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.434138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.434133 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.434237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.434152 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.508572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.508535 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:22:10.535954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.535899 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bcg\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536080 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536138 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536169 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536187 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536205 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536241 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536283 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536317 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536370 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536408 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536452 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536486 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536528 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536709 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536741 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536886 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdxq\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.536962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.536940 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.537385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.537043 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.537385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.537106 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.537498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.537412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.537964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.537527 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.537964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.537889 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.539650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.539622 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.540502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.540475 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.589762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.589733 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:22:10.599931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.599902 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.600539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.600514 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdxq\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.638436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638347 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638401 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638458 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bcg\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638541 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638602 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638634 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.638864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638726 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.639125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.639100 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.639397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.639378 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.639817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.639790 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.640483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.640457 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.641326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.638756 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.641494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.641370 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.641721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.641697 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.641793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.641725 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.642656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.642632 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.684352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.684323 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:10.689818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.689792 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:10.702064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.702034 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.702155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.702068 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bcg\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.735952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.735920 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:10.746302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.742789 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\" (UniqueName: \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\") pod \"68ecb737-5d04-414b-af48-13250ec0f214\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " Nov 01 05:22:10.746302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.744742 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\") pod \"68ecb737-5d04-414b-af48-13250ec0f214\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " Nov 01 05:22:10.746302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.745328 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\") pod \"68ecb737-5d04-414b-af48-13250ec0f214\" (UID: \"68ecb737-5d04-414b-af48-13250ec0f214\") " Nov 01 05:22:10.746302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.745738 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca" pod "68ecb737-5d04-414b-af48-13250ec0f214" (UID: "68ecb737-5d04-414b-af48-13250ec0f214"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:10.747031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.746706 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:10.751084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.751005 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca" pod "68ecb737-5d04-414b-af48-13250ec0f214" (UID: "68ecb737-5d04-414b-af48-13250ec0f214"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:10.773438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.753992 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr" pod "68ecb737-5d04-414b-af48-13250ec0f214" (UID: "68ecb737-5d04-414b-af48-13250ec0f214"). InnerVolumeSpecName "kube-api-access-ktcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:10.848772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.848737 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktcgr\" (UniqueName: \"kubernetes.io/projected/68ecb737-5d04-414b-af48-13250ec0f214-kube-api-access-ktcgr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:10.848772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:10.848774 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ecb737-5d04-414b-af48-13250ec0f214-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:11.125242 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:11.125215 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3fe719_9c8b_4773_99db_72a3027106b1.slice/crio-2de6b7ba2a7f71d14ebefcc3cdd98eb25349d83c0032ec0a6e3a92bcc289b042 WatchSource:0}: Error finding container 2de6b7ba2a7f71d14ebefcc3cdd98eb25349d83c0032ec0a6e3a92bcc289b042: Status 404 returned error can't find the container with id 2de6b7ba2a7f71d14ebefcc3cdd98eb25349d83c0032ec0a6e3a92bcc289b042 Nov 01 05:22:11.126614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.126587 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:22:11.128239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.128202 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.128340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.128327 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.128384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.128361 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.179322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.179294 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:22:11.184505 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:11.184475 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c6daa3_cb09_4c44_ae95_582f3c6e134a.slice/crio-131ec9ed29b5e6310f33e6624945cc8a1679d13f408705eae872c57a1f3d2485 WatchSource:0}: Error finding container 131ec9ed29b5e6310f33e6624945cc8a1679d13f408705eae872c57a1f3d2485: Status 404 returned error can't find the container with id 131ec9ed29b5e6310f33e6624945cc8a1679d13f408705eae872c57a1f3d2485 Nov 01 05:22:11.187211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.187178 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.187356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.187245 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.187356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.187301 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:11.551773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.551668 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" event={"ID":"68ecb737-5d04-414b-af48-13250ec0f214","Type":"ContainerDied","Data":"c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d"} Nov 01 05:22:11.552389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.552346 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a294d38119ad7bc40e0a6ee3b23b1428836a8907af5c8e8ec5594283a1513d" Nov 01 05:22:11.552389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.551704 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr" Nov 01 05:22:11.554761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.554720 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" event={"ID":"38c6daa3-cb09-4c44-ae95-582f3c6e134a","Type":"ContainerStarted","Data":"d49cc11b901a80a619d96edafd8a3fb65c11bb465676ceee857d11f7931c151a"} Nov 01 05:22:11.554892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.554791 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" event={"ID":"38c6daa3-cb09-4c44-ae95-582f3c6e134a","Type":"ContainerStarted","Data":"131ec9ed29b5e6310f33e6624945cc8a1679d13f408705eae872c57a1f3d2485"} Nov 01 05:22:11.557522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.557492 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" event={"ID":"ec3fe719-9c8b-4773-99db-72a3027106b1","Type":"ContainerStarted","Data":"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722"} Nov 01 05:22:11.557653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.557550 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" event={"ID":"ec3fe719-9c8b-4773-99db-72a3027106b1","Type":"ContainerStarted","Data":"2de6b7ba2a7f71d14ebefcc3cdd98eb25349d83c0032ec0a6e3a92bcc289b042"} Nov 01 05:22:11.682847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.682793 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podStartSLOduration=1.6827779550000002 podStartE2EDuration="1.682777955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:11.67858424 +0000 UTC m=+1146.819173160" watchObservedRunningTime="2026-11-01 05:22:11.682777955 +0000 UTC m=+1146.823366874" Nov 01 05:22:11.684887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.684865 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:11.736707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.736678 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:11.805978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:11.805876 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podStartSLOduration=1.8058613289999998 podStartE2EDuration="1.805861329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:11.802274353 +0000 UTC m=+1146.942863270" watchObservedRunningTime="2026-11-01 05:22:11.805861329 +0000 UTC m=+1146.946450247" Nov 01 05:22:12.693281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:12.693116 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:12.693281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:12.693182 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:12.737922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:12.737887 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:12.738079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:12.737951 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:13.693844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:13.693809 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:13.694510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:13.694477 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:13.736582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:13.736548 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:13.736765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:13.736608 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:14.692824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:14.692790 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:14.693118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:14.693079 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:14.737640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:14.737606 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:14.738167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:14.738140 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:15.684832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:15.684794 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:15.685143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:15.685109 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:15.737828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:15.737796 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:15.738403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:15.738372 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:16.703281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:16.687203 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:16.703281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:16.687294 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:16.751282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:16.747961 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:16.751282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:16.748017 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:17.697906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:17.690383 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:17.697906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:17.690445 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:17.736545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:17.736514 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:17.736766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:17.736565 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:18.685784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:18.685746 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 01 05:22:18.686171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:18.685815 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 01 05:22:18.737343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:18.737296 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 01 05:22:18.737512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:18.737372 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 01 05:22:19.690913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.690880 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:19.691476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.691278 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:19.692069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.692053 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:22:19.740538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.740512 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:19.740822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.740803 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:19.741567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:19.741547 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:22:20.543452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.543419 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4"] Nov 01 05:22:20.544009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.543995 4425 cpu_manager.go:414] "containerMap: removing container" podUID="68ecb737-5d04-414b-af48-13250ec0f214" containerName="revoke-keys" Nov 01 05:22:20.544058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.544010 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ecb737-5d04-414b-af48-13250ec0f214" containerName="revoke-keys" Nov 01 05:22:20.544113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.544105 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ecb737-5d04-414b-af48-13250ec0f214" containerName="revoke-keys" Nov 01 05:22:20.547889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.547872 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.575771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.575739 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.575937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.575791 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk56v\" (UniqueName: \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.576014 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.575995 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.602432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.602401 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:20.602592 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.602544 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:20.602655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.602615 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:20.602717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.602662 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:20.606531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.606503 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4"] Nov 01 05:22:20.676849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.676817 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.677043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.676913 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.677043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.676953 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk56v\" (UniqueName: \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.677631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.677602 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.679348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.679331 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.739596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.739558 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk56v\" (UniqueName: \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\") pod \"maas-api-revoke-sub-0345facf-zxpq4\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:20.861852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:20.861821 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:21.278671 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:21.278631 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253e0d15_3c47_4e0e_a213_f3c866384ab2.slice/crio-53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3 WatchSource:0}: Error finding container 53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3: Status 404 returned error can't find the container with id 53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3 Nov 01 05:22:21.280387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:21.280358 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4"] Nov 01 05:22:21.787729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:21.787680 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" event={"ID":"253e0d15-3c47-4e0e-a213-f3c866384ab2","Type":"ContainerStarted","Data":"ba296c5729933e1a9dfa1511772f0e84d6f389e6c4d15174e163d3c1c3d5da56"} Nov 01 05:22:21.788389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:21.788342 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" event={"ID":"253e0d15-3c47-4e0e-a213-f3c866384ab2","Type":"ContainerStarted","Data":"53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3"} Nov 01 05:22:21.909579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:21.909520 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" podStartSLOduration=1.9095066790000002 podStartE2EDuration="1.909506679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:21.908618244 +0000 UTC m=+1157.049207174" watchObservedRunningTime="2026-11-01 05:22:21.909506679 +0000 UTC m=+1157.050095597" Nov 01 05:22:22.796623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:22.796588 4425 generic.go:356] "Generic (PLEG): container finished" podID="253e0d15-3c47-4e0e-a213-f3c866384ab2" containerID="ba296c5729933e1a9dfa1511772f0e84d6f389e6c4d15174e163d3c1c3d5da56" exitCode=0 Nov 01 05:22:22.797033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:22.796672 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" event={"ID":"253e0d15-3c47-4e0e-a213-f3c866384ab2","Type":"ContainerDied","Data":"ba296c5729933e1a9dfa1511772f0e84d6f389e6c4d15174e163d3c1c3d5da56"} Nov 01 05:22:23.453050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.453018 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc"] Nov 01 05:22:23.457230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.457214 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.517774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.517744 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc"] Nov 01 05:22:23.540481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.540451 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.540641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.540511 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.540697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.540637 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjdz\" (UniqueName: \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.641845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.641808 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.642045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.641861 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.642045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.641889 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjdz\" (UniqueName: \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.642670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.642641 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.644486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.644463 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.705083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.705007 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjdz\" (UniqueName: \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\") pod \"maas-api-revoke-sub-8b1b6b51-dsrbc\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.771070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.771036 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:23.967417 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:23.967354 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:24.049428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.049396 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\") pod \"253e0d15-3c47-4e0e-a213-f3c866384ab2\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " Nov 01 05:22:24.049620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.049480 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\" (UniqueName: \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\") pod \"253e0d15-3c47-4e0e-a213-f3c866384ab2\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " Nov 01 05:22:24.049620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.049528 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\") pod \"253e0d15-3c47-4e0e-a213-f3c866384ab2\" (UID: \"253e0d15-3c47-4e0e-a213-f3c866384ab2\") " Nov 01 05:22:24.049965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.049934 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca" pod "253e0d15-3c47-4e0e-a213-f3c866384ab2" (UID: "253e0d15-3c47-4e0e-a213-f3c866384ab2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:24.050200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.050184 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:24.051569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.051547 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca" pod "253e0d15-3c47-4e0e-a213-f3c866384ab2" (UID: "253e0d15-3c47-4e0e-a213-f3c866384ab2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:24.051672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.051648 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v" pod "253e0d15-3c47-4e0e-a213-f3c866384ab2" (UID: "253e0d15-3c47-4e0e-a213-f3c866384ab2"). InnerVolumeSpecName "kube-api-access-xk56v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:24.151191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.151149 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253e0d15-3c47-4e0e-a213-f3c866384ab2-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:24.151191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.151183 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk56v\" (UniqueName: \"kubernetes.io/projected/253e0d15-3c47-4e0e-a213-f3c866384ab2-kube-api-access-xk56v\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:24.201972 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:24.201927 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735de069_0a10_4c8e_8aea_8349e53ba1c0.slice/crio-f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb WatchSource:0}: Error finding container f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb: Status 404 returned error can't find the container with id f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb Nov 01 05:22:24.203328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.203303 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc"] Nov 01 05:22:24.820709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.820673 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" event={"ID":"735de069-0a10-4c8e-8aea-8349e53ba1c0","Type":"ContainerStarted","Data":"1d02235a1763cc0dd3ef7663314b945473d66d56e838d2b253baa240d15de9b6"} Nov 01 05:22:24.820901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.820742 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" event={"ID":"735de069-0a10-4c8e-8aea-8349e53ba1c0","Type":"ContainerStarted","Data":"f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb"} Nov 01 05:22:24.822151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.822127 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" Nov 01 05:22:24.822151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.822135 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4" event={"ID":"253e0d15-3c47-4e0e-a213-f3c866384ab2","Type":"ContainerDied","Data":"53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3"} Nov 01 05:22:24.822333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.822178 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53efc4150fc812a337e3c9af5488d6b8b50f962e60c534e0803a5f4a6ba710f3" Nov 01 05:22:24.945732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:24.945624 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" podStartSLOduration=1.945607981 podStartE2EDuration="1.945607981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:24.938466058 +0000 UTC m=+1160.079054996" watchObservedRunningTime="2026-11-01 05:22:24.945607981 +0000 UTC m=+1160.086196966" Nov 01 05:22:25.831463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:25.831427 4425 generic.go:356] "Generic (PLEG): container finished" podID="735de069-0a10-4c8e-8aea-8349e53ba1c0" containerID="1d02235a1763cc0dd3ef7663314b945473d66d56e838d2b253baa240d15de9b6" exitCode=0 Nov 01 05:22:25.831906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:25.831519 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" event={"ID":"735de069-0a10-4c8e-8aea-8349e53ba1c0","Type":"ContainerDied","Data":"1d02235a1763cc0dd3ef7663314b945473d66d56e838d2b253baa240d15de9b6"} Nov 01 05:22:26.955743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:22:26.955708 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:22:26.990396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:26.990369 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:27.089329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.089294 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\") pod \"735de069-0a10-4c8e-8aea-8349e53ba1c0\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " Nov 01 05:22:27.089517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.089352 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\" (UniqueName: \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\") pod \"735de069-0a10-4c8e-8aea-8349e53ba1c0\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " Nov 01 05:22:27.089517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.089406 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\") pod \"735de069-0a10-4c8e-8aea-8349e53ba1c0\" (UID: \"735de069-0a10-4c8e-8aea-8349e53ba1c0\") " Nov 01 05:22:27.089874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.089816 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca" pod "735de069-0a10-4c8e-8aea-8349e53ba1c0" (UID: "735de069-0a10-4c8e-8aea-8349e53ba1c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:27.091520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.091499 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca" pod "735de069-0a10-4c8e-8aea-8349e53ba1c0" (UID: "735de069-0a10-4c8e-8aea-8349e53ba1c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:27.091665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.091651 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz" pod "735de069-0a10-4c8e-8aea-8349e53ba1c0" (UID: "735de069-0a10-4c8e-8aea-8349e53ba1c0"). InnerVolumeSpecName "kube-api-access-jxjdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:27.190682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.190649 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:27.190682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.190680 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjdz\" (UniqueName: \"kubernetes.io/projected/735de069-0a10-4c8e-8aea-8349e53ba1c0-kube-api-access-jxjdz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:27.190682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.190693 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735de069-0a10-4c8e-8aea-8349e53ba1c0-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:27.848008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.847975 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" Nov 01 05:22:27.848187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.848023 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc" event={"ID":"735de069-0a10-4c8e-8aea-8349e53ba1c0","Type":"ContainerDied","Data":"f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb"} Nov 01 05:22:27.848187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:27.848071 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f463327a2af7460b01a6ea75f5b47f4efd26c6f3d63ce624cc6f71ce3ea1d3bb" Nov 01 05:22:35.199834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.199783 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr"] Nov 01 05:22:35.201403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.201374 4425 cpu_manager.go:414] "containerMap: removing container" podUID="253e0d15-3c47-4e0e-a213-f3c866384ab2" containerName="revoke-keys" Nov 01 05:22:35.201589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.201572 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253e0d15-3c47-4e0e-a213-f3c866384ab2" containerName="revoke-keys" Nov 01 05:22:35.201773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.201759 4425 cpu_manager.go:414] "containerMap: removing container" podUID="735de069-0a10-4c8e-8aea-8349e53ba1c0" containerName="revoke-keys" Nov 01 05:22:35.201872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.201858 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735de069-0a10-4c8e-8aea-8349e53ba1c0" containerName="revoke-keys" Nov 01 05:22:35.202100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.202085 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="253e0d15-3c47-4e0e-a213-f3c866384ab2" containerName="revoke-keys" Nov 01 05:22:35.202236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.202223 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="735de069-0a10-4c8e-8aea-8349e53ba1c0" containerName="revoke-keys" Nov 01 05:22:35.208277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.207867 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.315907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.315746 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:35.315907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.315816 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:35.315907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.315874 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:35.351280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.337523 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:35.390282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.360950 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.390282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.361058 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dzt\" (UniqueName: \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.390282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.361220 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.521332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.510767 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.521332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.510910 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.521332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.510995 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dzt\" (UniqueName: \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.527431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.527398 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.558452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.558403 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.631283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.623697 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr"] Nov 01 05:22:35.681932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.666773 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dzt\" (UniqueName: \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\") pod \"maas-api-revoke-sub-3b253a1c-bz8rr\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:35.914511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:35.914473 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:36.830236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:36.829874 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr"] Nov 01 05:22:36.831055 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:36.831023 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178b7008_31b1_4f59_a219_282693739a7c.slice/crio-a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce WatchSource:0}: Error finding container a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce: Status 404 returned error can't find the container with id a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce Nov 01 05:22:37.015893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:37.015860 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" event={"ID":"178b7008-31b1-4f59-a219-282693739a7c","Type":"ContainerStarted","Data":"a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce"} Nov 01 05:22:37.056464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:37.056436 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr"] Nov 01 05:22:37.061663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:37.061640 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f56600-ts9fr"] Nov 01 05:22:38.024506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:38.024413 4425 generic.go:356] "Generic (PLEG): container finished" podID="178b7008-31b1-4f59-a219-282693739a7c" containerID="5966607157daf123b13d9a65bee830115c34359e9fb9ff199b23db3a7576f857" exitCode=0 Nov 01 05:22:38.024882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:38.024503 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" event={"ID":"178b7008-31b1-4f59-a219-282693739a7c","Type":"ContainerDied","Data":"5966607157daf123b13d9a65bee830115c34359e9fb9ff199b23db3a7576f857"} Nov 01 05:22:38.974227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:38.974189 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf" path="/var/lib/kubelet/pods/67fb1ef9-e05e-4d4f-84d7-8c281c39e3bf/volumes" Nov 01 05:22:39.175759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.175734 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:39.258772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.258699 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\") pod \"178b7008-31b1-4f59-a219-282693739a7c\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " Nov 01 05:22:39.258772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.258755 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\" (UniqueName: \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\") pod \"178b7008-31b1-4f59-a219-282693739a7c\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " Nov 01 05:22:39.258948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.258887 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\") pod \"178b7008-31b1-4f59-a219-282693739a7c\" (UID: \"178b7008-31b1-4f59-a219-282693739a7c\") " Nov 01 05:22:39.259140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.259112 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca" pod "178b7008-31b1-4f59-a219-282693739a7c" (UID: "178b7008-31b1-4f59-a219-282693739a7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:39.259459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.259431 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178b7008-31b1-4f59-a219-282693739a7c-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:39.261086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.261059 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt" pod "178b7008-31b1-4f59-a219-282693739a7c" (UID: "178b7008-31b1-4f59-a219-282693739a7c"). InnerVolumeSpecName "kube-api-access-j4dzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:39.261198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.261124 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca" pod "178b7008-31b1-4f59-a219-282693739a7c" (UID: "178b7008-31b1-4f59-a219-282693739a7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:39.360944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.360907 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178b7008-31b1-4f59-a219-282693739a7c-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:39.360944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.360940 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4dzt\" (UniqueName: \"kubernetes.io/projected/178b7008-31b1-4f59-a219-282693739a7c-kube-api-access-j4dzt\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:39.513649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.513577 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:22:39.514149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.514137 4425 cpu_manager.go:414] "containerMap: removing container" podUID="178b7008-31b1-4f59-a219-282693739a7c" containerName="revoke-keys" Nov 01 05:22:39.514187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.514151 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178b7008-31b1-4f59-a219-282693739a7c" containerName="revoke-keys" Nov 01 05:22:39.514290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.514280 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="178b7008-31b1-4f59-a219-282693739a7c" containerName="revoke-keys" Nov 01 05:22:39.517961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.517942 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.562968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.562926 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.562978 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563063 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563186 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6j6\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563245 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563353 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563440 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563518 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.563607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.563590 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.578451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.578424 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:22:39.665243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665206 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6j6\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665301 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665326 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665479 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665548 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665609 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665674 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665746 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665747 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665825 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.665993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.665914 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.666283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.666239 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.666416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.666355 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.668241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.668212 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.668439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.668422 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.727683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.727655 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.727683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.727675 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6j6\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-7gfxb\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.766724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.766649 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg"] Nov 01 05:22:39.772001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.771979 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.829504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.829476 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg"] Nov 01 05:22:39.831942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.831920 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:39.867706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.867675 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84d4\" (UniqueName: \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.867919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.867880 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.868097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.868003 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.969617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.969586 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.969779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.969658 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.969867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.969778 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84d4\" (UniqueName: \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.970306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.970287 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:39.972191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:39.972169 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:40.032443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.032378 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84d4\" (UniqueName: \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\") pod \"maas-api-revoke-sub-bc035ea1-zb7gg\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:40.039862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.039836 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" Nov 01 05:22:40.039985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.039859 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr" event={"ID":"178b7008-31b1-4f59-a219-282693739a7c","Type":"ContainerDied","Data":"a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce"} Nov 01 05:22:40.039985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.039908 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a357cdfceb22b05053efaa03412fd728827369d8062cfad8172301ef2de1ccce" Nov 01 05:22:40.087669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.087641 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:40.271801 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:40.271770 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda949816b_9a7e_4580_a380_73f1262abda3.slice/crio-98c7fdde10883a4a43f9e6faa3809f1e5d3d47a52803973160f0c78b3f5e07e7 WatchSource:0}: Error finding container 98c7fdde10883a4a43f9e6faa3809f1e5d3d47a52803973160f0c78b3f5e07e7: Status 404 returned error can't find the container with id 98c7fdde10883a4a43f9e6faa3809f1e5d3d47a52803973160f0c78b3f5e07e7 Nov 01 05:22:40.274988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.274961 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:22:40.275090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.275049 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:40.275133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.275118 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:40.275164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.275151 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:22:40.468477 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:40.468433 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b4da31_4330_432d_afe9_b10ab99fa14d.slice/crio-9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3 WatchSource:0}: Error finding container 9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3: Status 404 returned error can't find the container with id 9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3 Nov 01 05:22:40.468650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:40.468634 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg"] Nov 01 05:22:41.077217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.077172 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" event={"ID":"b3b4da31-4330-432d-afe9-b10ab99fa14d","Type":"ContainerStarted","Data":"3a40c45847da2483465a19273d3f4efac1b084012ec83fedfaaf37ccdfbfbbdc"} Nov 01 05:22:41.077425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.077239 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" event={"ID":"b3b4da31-4330-432d-afe9-b10ab99fa14d","Type":"ContainerStarted","Data":"9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3"} Nov 01 05:22:41.079065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.079034 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" event={"ID":"a949816b-9a7e-4580-a380-73f1262abda3","Type":"ContainerStarted","Data":"66612c6c6efcd6807a1d3cfb41e294988db17aa14b538e8711c639945b9ddbac"} Nov 01 05:22:41.079189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.079091 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" event={"ID":"a949816b-9a7e-4580-a380-73f1262abda3","Type":"ContainerStarted","Data":"98c7fdde10883a4a43f9e6faa3809f1e5d3d47a52803973160f0c78b3f5e07e7"} Nov 01 05:22:41.198199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.198150 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" podStartSLOduration=2.198136257 podStartE2EDuration="2.198136257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:41.195133633 +0000 UTC m=+1176.335722554" watchObservedRunningTime="2026-11-01 05:22:41.198136257 +0000 UTC m=+1176.338725229" Nov 01 05:22:41.333385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.333271 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podStartSLOduration=2.333245038 podStartE2EDuration="2.333245038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:41.329721152 +0000 UTC m=+1176.470310071" watchObservedRunningTime="2026-11-01 05:22:41.333245038 +0000 UTC m=+1176.473834027" Nov 01 05:22:41.832177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.832146 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:41.833371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.833316 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:41.833528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:41.833388 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:42.093944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:42.093858 4425 generic.go:356] "Generic (PLEG): container finished" podID="b3b4da31-4330-432d-afe9-b10ab99fa14d" containerID="3a40c45847da2483465a19273d3f4efac1b084012ec83fedfaaf37ccdfbfbbdc" exitCode=0 Nov 01 05:22:42.098747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:42.098229 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" event={"ID":"b3b4da31-4330-432d-afe9-b10ab99fa14d","Type":"ContainerDied","Data":"3a40c45847da2483465a19273d3f4efac1b084012ec83fedfaaf37ccdfbfbbdc"} Nov 01 05:22:42.835293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:42.832419 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:42.835293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:42.832469 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:43.327584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.327556 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:43.415904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.415818 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\" (UniqueName: \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\") pod \"b3b4da31-4330-432d-afe9-b10ab99fa14d\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " Nov 01 05:22:43.416049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.415940 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\") pod \"b3b4da31-4330-432d-afe9-b10ab99fa14d\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " Nov 01 05:22:43.416049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.416033 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\") pod \"b3b4da31-4330-432d-afe9-b10ab99fa14d\" (UID: \"b3b4da31-4330-432d-afe9-b10ab99fa14d\") " Nov 01 05:22:43.416409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.416378 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca" pod "b3b4da31-4330-432d-afe9-b10ab99fa14d" (UID: "b3b4da31-4330-432d-afe9-b10ab99fa14d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:43.418182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.418159 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4" pod "b3b4da31-4330-432d-afe9-b10ab99fa14d" (UID: "b3b4da31-4330-432d-afe9-b10ab99fa14d"). InnerVolumeSpecName "kube-api-access-j84d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:43.418182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.418171 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca" pod "b3b4da31-4330-432d-afe9-b10ab99fa14d" (UID: "b3b4da31-4330-432d-afe9-b10ab99fa14d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:43.517289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.517228 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:43.517289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.517289 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84d4\" (UniqueName: \"kubernetes.io/projected/b3b4da31-4330-432d-afe9-b10ab99fa14d-kube-api-access-j84d4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:43.517496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.517306 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3b4da31-4330-432d-afe9-b10ab99fa14d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:43.832690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.832641 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:43.832894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:43.832855 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:44.116721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:44.116630 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" Nov 01 05:22:44.116721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:44.116658 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg" event={"ID":"b3b4da31-4330-432d-afe9-b10ab99fa14d","Type":"ContainerDied","Data":"9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3"} Nov 01 05:22:44.116721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:44.116721 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a388356d24e187e7267b5a85195c55ec4ea2c4f62c174dbcbb33fee6a05fda3" Nov 01 05:22:44.832793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:44.832746 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:44.832970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:44.832841 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:45.832315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:45.832243 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:45.832698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:45.832351 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:46.323893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:46.323857 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:22:46.832364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:46.832325 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 05:22:46.832905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:46.832398 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 05:22:47.909563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:47.909535 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:47.910468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:47.910444 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:47.912817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:47.912793 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:22:50.796653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:50.796607 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g"] Nov 01 05:22:50.798573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:50.798536 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b3b4da31-4330-432d-afe9-b10ab99fa14d" containerName="revoke-keys" Nov 01 05:22:50.798746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:50.798731 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b4da31-4330-432d-afe9-b10ab99fa14d" containerName="revoke-keys" Nov 01 05:22:50.812277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:50.799175 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b4da31-4330-432d-afe9-b10ab99fa14d" containerName="revoke-keys" Nov 01 05:22:50.950308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:50.943580 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.084128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.084091 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.084458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.084435 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.084672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.084655 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g69t\" (UniqueName: \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.117291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.088622 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:51.117291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.099393 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:51.117291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.106539 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:51.117291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.106668 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:51.213291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.211042 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.213291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.211136 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.213291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.212683 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.242279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.229421 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g"] Nov 01 05:22:51.242279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.236668 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g69t\" (UniqueName: \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.261277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.260686 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.307279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.307057 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g69t\" (UniqueName: \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\") pod \"maas-api-revoke-sub-821ecd17-wbh8g\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.329280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.325724 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" containerID="cri-o://938d2c016ceb1c28dc11da8c826c24fefcffecbc49d669c21a66571aadd9305c" gracePeriod=25 Nov 01 05:22:51.529102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.529009 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:51.974218 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:51.974189 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ba969c_c314_4c1b_8125_e64a722efc5d.slice/crio-cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e WatchSource:0}: Error finding container cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e: Status 404 returned error can't find the container with id cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e Nov 01 05:22:51.976654 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:51.976629 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g"] Nov 01 05:22:52.273202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:52.273170 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" event={"ID":"b4ba969c-c314-4c1b-8125-e64a722efc5d","Type":"ContainerStarted","Data":"697928dc385f065633009db522d7c805685d6b0e68931fa16de7e4cd6e20c3de"} Nov 01 05:22:52.273350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:52.273230 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" event={"ID":"b4ba969c-c314-4c1b-8125-e64a722efc5d","Type":"ContainerStarted","Data":"cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e"} Nov 01 05:22:52.395421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:52.395371 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" podStartSLOduration=2.3953579830000002 podStartE2EDuration="2.395357983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:52.391446771 +0000 UTC m=+1187.532035690" watchObservedRunningTime="2026-11-01 05:22:52.395357983 +0000 UTC m=+1187.535946908" Nov 01 05:22:53.281499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:53.281415 4425 generic.go:356] "Generic (PLEG): container finished" podID="b4ba969c-c314-4c1b-8125-e64a722efc5d" containerID="697928dc385f065633009db522d7c805685d6b0e68931fa16de7e4cd6e20c3de" exitCode=0 Nov 01 05:22:53.281876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:53.281495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" event={"ID":"b4ba969c-c314-4c1b-8125-e64a722efc5d","Type":"ContainerDied","Data":"697928dc385f065633009db522d7c805685d6b0e68931fa16de7e4cd6e20c3de"} Nov 01 05:22:54.443448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.443417 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:54.615444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.615414 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\") pod \"b4ba969c-c314-4c1b-8125-e64a722efc5d\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " Nov 01 05:22:54.615625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.615547 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\" (UniqueName: \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\") pod \"b4ba969c-c314-4c1b-8125-e64a722efc5d\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " Nov 01 05:22:54.615625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.615568 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\") pod \"b4ba969c-c314-4c1b-8125-e64a722efc5d\" (UID: \"b4ba969c-c314-4c1b-8125-e64a722efc5d\") " Nov 01 05:22:54.615802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.615777 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca" pod "b4ba969c-c314-4c1b-8125-e64a722efc5d" (UID: "b4ba969c-c314-4c1b-8125-e64a722efc5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:54.616033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.616019 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:54.617788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.617758 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t" pod "b4ba969c-c314-4c1b-8125-e64a722efc5d" (UID: "b4ba969c-c314-4c1b-8125-e64a722efc5d"). InnerVolumeSpecName "kube-api-access-6g69t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:54.617900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.617833 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca" pod "b4ba969c-c314-4c1b-8125-e64a722efc5d" (UID: "b4ba969c-c314-4c1b-8125-e64a722efc5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:54.717176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.717144 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g69t\" (UniqueName: \"kubernetes.io/projected/b4ba969c-c314-4c1b-8125-e64a722efc5d-kube-api-access-6g69t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:54.717176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:54.717172 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ba969c-c314-4c1b-8125-e64a722efc5d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:55.297932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:55.297852 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" Nov 01 05:22:55.297932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:55.297884 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g" event={"ID":"b4ba969c-c314-4c1b-8125-e64a722efc5d","Type":"ContainerDied","Data":"cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e"} Nov 01 05:22:55.297932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:55.297932 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdbed9895914d2a8bbfd65b19103382fea6f4fe9ec75e4351985d7765a907a4e" Nov 01 05:22:56.049915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.049881 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l"] Nov 01 05:22:56.101021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.100985 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50e6159-sdp8l"] Nov 01 05:22:56.606562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.606532 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9"] Nov 01 05:22:56.607106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.607093 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4ba969c-c314-4c1b-8125-e64a722efc5d" containerName="revoke-keys" Nov 01 05:22:56.607171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.607107 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ba969c-c314-4c1b-8125-e64a722efc5d" containerName="revoke-keys" Nov 01 05:22:56.608171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.608144 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ba969c-c314-4c1b-8125-e64a722efc5d" containerName="revoke-keys" Nov 01 05:22:56.613790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.613759 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.669021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.668990 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9"] Nov 01 05:22:56.713697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.713668 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:22:56.713697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.713695 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:22:56.713907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.713695 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:22:56.718739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.718720 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:22:56.743420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.743391 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqjr\" (UniqueName: \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.743575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.743518 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.743575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.743549 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.845782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.845748 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqjr\" (UniqueName: \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.845987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.845861 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.845987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.845891 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.846620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.846585 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.848332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.848308 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.908357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.908283 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqjr\" (UniqueName: \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\") pod \"maas-api-revoke-sub-8dff679c-dx8j9\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.930749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.930723 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:56.952556 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:22:56.952524 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:22:56.976135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:56.976095 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d367029-df40-4140-a20c-1aec0a3db4bd" path="/var/lib/kubelet/pods/1d367029-df40-4140-a20c-1aec0a3db4bd/volumes" Nov 01 05:22:57.315565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:57.315541 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9"] Nov 01 05:22:57.315929 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:22:57.315716 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7ae036_6883_4b89_b433_17b9e6addd63.slice/crio-3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0 WatchSource:0}: Error finding container 3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0: Status 404 returned error can't find the container with id 3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0 Nov 01 05:22:58.321230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:58.321193 4425 generic.go:356] "Generic (PLEG): container finished" podID="5f7ae036-6883-4b89-b433-17b9e6addd63" containerID="50d5548829f3ac911f3087a8162d16857116040ac13ad974ffd38a27ef7dbcd9" exitCode=0 Nov 01 05:22:58.321632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:58.321302 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" event={"ID":"5f7ae036-6883-4b89-b433-17b9e6addd63","Type":"ContainerDied","Data":"50d5548829f3ac911f3087a8162d16857116040ac13ad974ffd38a27ef7dbcd9"} Nov 01 05:22:58.321632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:58.321354 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" event={"ID":"5f7ae036-6883-4b89-b433-17b9e6addd63","Type":"ContainerStarted","Data":"3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0"} Nov 01 05:22:58.571245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:58.571201 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:22:59.477461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.477433 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:22:59.573826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.573789 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\") pod \"5f7ae036-6883-4b89-b433-17b9e6addd63\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " Nov 01 05:22:59.574007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.573888 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\" (UniqueName: \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\") pod \"5f7ae036-6883-4b89-b433-17b9e6addd63\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " Nov 01 05:22:59.574007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.573989 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\") pod \"5f7ae036-6883-4b89-b433-17b9e6addd63\" (UID: \"5f7ae036-6883-4b89-b433-17b9e6addd63\") " Nov 01 05:22:59.574426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.574360 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca" pod "5f7ae036-6883-4b89-b433-17b9e6addd63" (UID: "5f7ae036-6883-4b89-b433-17b9e6addd63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:22:59.574825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.574803 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:59.576147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.576119 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca" pod "5f7ae036-6883-4b89-b433-17b9e6addd63" (UID: "5f7ae036-6883-4b89-b433-17b9e6addd63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:22:59.576147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.576142 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr" pod "5f7ae036-6883-4b89-b433-17b9e6addd63" (UID: "5f7ae036-6883-4b89-b433-17b9e6addd63"). InnerVolumeSpecName "kube-api-access-ffqjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:22:59.675723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.675685 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7ae036-6883-4b89-b433-17b9e6addd63-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:22:59.675723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:22:59.675717 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqjr\" (UniqueName: \"kubernetes.io/projected/5f7ae036-6883-4b89-b433-17b9e6addd63-kube-api-access-ffqjr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:00.339856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:00.339824 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" Nov 01 05:23:00.339856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:00.339845 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9" event={"ID":"5f7ae036-6883-4b89-b433-17b9e6addd63","Type":"ContainerDied","Data":"3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0"} Nov 01 05:23:00.340123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:00.339901 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3399daa85a11a60179a462f0490aa95cb24af6586948ec94f0806dcaf24770c0" Nov 01 05:23:07.922761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.922720 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk"] Nov 01 05:23:07.923579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.923557 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5f7ae036-6883-4b89-b433-17b9e6addd63" containerName="revoke-keys" Nov 01 05:23:07.923579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.923579 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7ae036-6883-4b89-b433-17b9e6addd63" containerName="revoke-keys" Nov 01 05:23:07.923753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.923731 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7ae036-6883-4b89-b433-17b9e6addd63" containerName="revoke-keys" Nov 01 05:23:07.928558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.928536 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:07.972235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.972208 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:07.972454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.972430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:07.972533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.972472 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msjp\" (UniqueName: \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:07.982630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.982602 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:23:07.982743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.982693 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:23:07.982743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.982693 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:23:07.982840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.982777 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:23:07.984009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:07.983986 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk"] Nov 01 05:23:08.073832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.073798 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.074031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.073917 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.074031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.073938 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msjp\" (UniqueName: \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.074685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.074661 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.076450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.076429 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.136992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.136953 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msjp\" (UniqueName: \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\") pod \"maas-api-revoke-sub-ef0173bc-f2vmk\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.242571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.242492 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:08.571086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.571057 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:23:08.621439 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:08.621407 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70783ac9_9aac_4ba5_a941_00c25c4f68ce.slice/crio-dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36 WatchSource:0}: Error finding container dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36: Status 404 returned error can't find the container with id dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36 Nov 01 05:23:08.622554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:08.622531 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk"] Nov 01 05:23:09.409545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:09.409492 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" event={"ID":"70783ac9-9aac-4ba5-a941-00c25c4f68ce","Type":"ContainerStarted","Data":"3d51e4076a7c8c705766d7990ab8edfa7b60514482d6d22bebfba6073f7fbdf8"} Nov 01 05:23:09.409995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:09.409592 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" event={"ID":"70783ac9-9aac-4ba5-a941-00c25c4f68ce","Type":"ContainerStarted","Data":"dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36"} Nov 01 05:23:09.530094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:09.530045 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" podStartSLOduration=2.530031669 podStartE2EDuration="2.530031669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:23:09.527390257 +0000 UTC m=+1204.667979195" watchObservedRunningTime="2026-11-01 05:23:09.530031669 +0000 UTC m=+1204.670620588" Nov 01 05:23:10.449639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:10.449602 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" event={"ID":"70783ac9-9aac-4ba5-a941-00c25c4f68ce","Type":"ContainerDied","Data":"3d51e4076a7c8c705766d7990ab8edfa7b60514482d6d22bebfba6073f7fbdf8"} Nov 01 05:23:10.455629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:10.455595 4425 generic.go:356] "Generic (PLEG): container finished" podID="70783ac9-9aac-4ba5-a941-00c25c4f68ce" containerID="3d51e4076a7c8c705766d7990ab8edfa7b60514482d6d22bebfba6073f7fbdf8" exitCode=0 Nov 01 05:23:11.534300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:11.516806 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:11.586162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:11.586124 4425 generic.go:356] "Generic (PLEG): container finished" podID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerID="938d2c016ceb1c28dc11da8c826c24fefcffecbc49d669c21a66571aadd9305c" exitCode=0 Nov 01 05:23:11.602427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:11.602378 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" event={"ID":"999804dc-8517-448d-a6bd-6528f0b6fd84","Type":"ContainerDied","Data":"938d2c016ceb1c28dc11da8c826c24fefcffecbc49d669c21a66571aadd9305c"} Nov 01 05:23:11.698405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:11.698347 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 05:23:12.145897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.143359 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg"] Nov 01 05:23:12.312992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.312943 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef4940a-pbdxg"] Nov 01 05:23:12.405596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.405568 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:23:12.454315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.454286 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\") pod \"999804dc-8517-448d-a6bd-6528f0b6fd84\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " Nov 01 05:23:12.454492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.454341 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\" (UniqueName: \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\") pod \"999804dc-8517-448d-a6bd-6528f0b6fd84\" (UID: \"999804dc-8517-448d-a6bd-6528f0b6fd84\") " Nov 01 05:23:12.454767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.454742 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume" pod "999804dc-8517-448d-a6bd-6528f0b6fd84" (UID: "999804dc-8517-448d-a6bd-6528f0b6fd84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:12.454924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.454908 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999804dc-8517-448d-a6bd-6528f0b6fd84-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:12.456816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.456781 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8" pod "999804dc-8517-448d-a6bd-6528f0b6fd84" (UID: "999804dc-8517-448d-a6bd-6528f0b6fd84"). InnerVolumeSpecName "kube-api-access-dbhg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:12.489980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.489957 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:12.555783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.555747 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\") pod \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " Nov 01 05:23:12.556309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.555915 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\") pod \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " Nov 01 05:23:12.556309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.556016 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\" (UniqueName: \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\") pod \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\" (UID: \"70783ac9-9aac-4ba5-a941-00c25c4f68ce\") " Nov 01 05:23:12.556470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.556435 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca" pod "70783ac9-9aac-4ba5-a941-00c25c4f68ce" (UID: "70783ac9-9aac-4ba5-a941-00c25c4f68ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:12.556673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.556653 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbhg8\" (UniqueName: \"kubernetes.io/projected/999804dc-8517-448d-a6bd-6528f0b6fd84-kube-api-access-dbhg8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:12.556673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.556674 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:12.558725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.558694 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp" pod "70783ac9-9aac-4ba5-a941-00c25c4f68ce" (UID: "70783ac9-9aac-4ba5-a941-00c25c4f68ce"). InnerVolumeSpecName "kube-api-access-9msjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:12.558875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.558852 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca" pod "70783ac9-9aac-4ba5-a941-00c25c4f68ce" (UID: "70783ac9-9aac-4ba5-a941-00c25c4f68ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:12.600336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.600307 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" Nov 01 05:23:12.600336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.600317 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk" event={"ID":"70783ac9-9aac-4ba5-a941-00c25c4f68ce","Type":"ContainerDied","Data":"dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36"} Nov 01 05:23:12.600538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.600376 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc0a48195061ce1492955840dbd123fc60975904396afce2697b33b5c2dfc36" Nov 01 05:23:12.602077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.602052 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" event={"ID":"999804dc-8517-448d-a6bd-6528f0b6fd84","Type":"ContainerDied","Data":"542dbfc4caa04f7f54412283f8c9cc0a007cf0d57837b722fa976e451ffc5d4d"} Nov 01 05:23:12.602190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.602085 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m" Nov 01 05:23:12.602190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.602114 4425 scope.go:122] "RemoveContainer" containerID="938d2c016ceb1c28dc11da8c826c24fefcffecbc49d669c21a66571aadd9305c" Nov 01 05:23:12.657810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.657776 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70783ac9-9aac-4ba5-a941-00c25c4f68ce-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:12.657810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.657812 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msjp\" (UniqueName: \"kubernetes.io/projected/70783ac9-9aac-4ba5-a941-00c25c4f68ce-kube-api-access-9msjp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:12.814006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.813972 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:23:12.820568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.820539 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ea125-595686979-2wc5m"] Nov 01 05:23:12.975504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.975428 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" path="/var/lib/kubelet/pods/999804dc-8517-448d-a6bd-6528f0b6fd84/volumes" Nov 01 05:23:12.975877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:12.975864 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2b998c-e641-4a9d-96fe-0d90911dc70e" path="/var/lib/kubelet/pods/9b2b998c-e641-4a9d-96fe-0d90911dc70e/volumes" Nov 01 05:23:14.050366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:14.050330 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59"] Nov 01 05:23:14.078448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:14.078418 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237486cd-c4g59"] Nov 01 05:23:14.974600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:14.974568 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694fe92c-a8f4-4f35-93bd-9ec7a73061ef" path="/var/lib/kubelet/pods/694fe92c-a8f4-4f35-93bd-9ec7a73061ef/volumes" Nov 01 05:23:18.667333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667261 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz"] Nov 01 05:23:18.667890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667875 4425 cpu_manager.go:414] "containerMap: removing container" podUID="70783ac9-9aac-4ba5-a941-00c25c4f68ce" containerName="revoke-keys" Nov 01 05:23:18.667929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667892 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70783ac9-9aac-4ba5-a941-00c25c4f68ce" containerName="revoke-keys" Nov 01 05:23:18.667929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667907 4425 cpu_manager.go:414] "containerMap: removing container" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" Nov 01 05:23:18.667929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667912 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" Nov 01 05:23:18.668024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.667996 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="70783ac9-9aac-4ba5-a941-00c25c4f68ce" containerName="revoke-keys" Nov 01 05:23:18.668024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.668006 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="999804dc-8517-448d-a6bd-6528f0b6fd84" containerName="payload-pre-processing" Nov 01 05:23:18.677671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.677646 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.729046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.729010 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz"] Nov 01 05:23:18.775281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.775234 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:23:18.775281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.775234 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:23:18.775483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.775295 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:23:18.775483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.775328 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:23:18.836092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.836051 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ct9t\" (UniqueName: \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.836092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.836100 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.836337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.836227 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.938010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.937920 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ct9t\" (UniqueName: \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.938010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.937969 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.938311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.938014 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.938692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.938671 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:18.940556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:18.940537 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:19.000736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:19.000703 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ct9t\" (UniqueName: \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\") pod \"maas-api-revoke-sub-e3ed1d2d-hhlwz\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:19.292012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:19.291928 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:19.670933 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:19.670903 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e36600_22d7_49c7_8cd0_903dce78e7d3.slice/crio-262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0 WatchSource:0}: Error finding container 262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0: Status 404 returned error can't find the container with id 262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0 Nov 01 05:23:19.672463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:19.672436 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz"] Nov 01 05:23:20.668351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:20.668320 4425 generic.go:356] "Generic (PLEG): container finished" podID="00e36600-22d7-49c7-8cd0-903dce78e7d3" containerID="2a262b875850c3298303f89805d4e387ccee337c24ddef9716436c9c4a03c994" exitCode=0 Nov 01 05:23:20.668504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:20.668377 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" event={"ID":"00e36600-22d7-49c7-8cd0-903dce78e7d3","Type":"ContainerDied","Data":"2a262b875850c3298303f89805d4e387ccee337c24ddef9716436c9c4a03c994"} Nov 01 05:23:20.668504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:20.668442 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" event={"ID":"00e36600-22d7-49c7-8cd0-903dce78e7d3","Type":"ContainerStarted","Data":"262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0"} Nov 01 05:23:21.832432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.832405 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:21.978706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.978614 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\" (UniqueName: \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\") pod \"00e36600-22d7-49c7-8cd0-903dce78e7d3\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " Nov 01 05:23:21.978864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.978771 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\") pod \"00e36600-22d7-49c7-8cd0-903dce78e7d3\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " Nov 01 05:23:21.978864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.978799 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\") pod \"00e36600-22d7-49c7-8cd0-903dce78e7d3\" (UID: \"00e36600-22d7-49c7-8cd0-903dce78e7d3\") " Nov 01 05:23:21.979292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.979231 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca" pod "00e36600-22d7-49c7-8cd0-903dce78e7d3" (UID: "00e36600-22d7-49c7-8cd0-903dce78e7d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:21.981011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.980984 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t" pod "00e36600-22d7-49c7-8cd0-903dce78e7d3" (UID: "00e36600-22d7-49c7-8cd0-903dce78e7d3"). InnerVolumeSpecName "kube-api-access-7ct9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:21.981011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:21.980997 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca" pod "00e36600-22d7-49c7-8cd0-903dce78e7d3" (UID: "00e36600-22d7-49c7-8cd0-903dce78e7d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:22.080461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.080423 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:22.080461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.080458 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e36600-22d7-49c7-8cd0-903dce78e7d3-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:22.080461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.080470 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ct9t\" (UniqueName: \"kubernetes.io/projected/00e36600-22d7-49c7-8cd0-903dce78e7d3-kube-api-access-7ct9t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:22.684017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.683987 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" Nov 01 05:23:22.684017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.683998 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz" event={"ID":"00e36600-22d7-49c7-8cd0-903dce78e7d3","Type":"ContainerDied","Data":"262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0"} Nov 01 05:23:22.684272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:22.684084 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262882169cb551668f7e5aa61a583a3427faac0b237b83629dd2ababa1d369b0" Nov 01 05:23:26.997498 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:23:26.993212 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:23:28.356320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:28.350798 4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:28.703339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:28.694352 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:28.796884 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:28.784057 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:29.619476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.619443 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v"] Nov 01 05:23:29.620048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.620034 4425 cpu_manager.go:414] "containerMap: removing container" podUID="00e36600-22d7-49c7-8cd0-903dce78e7d3" containerName="revoke-keys" Nov 01 05:23:29.620048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.620048 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e36600-22d7-49c7-8cd0-903dce78e7d3" containerName="revoke-keys" Nov 01 05:23:29.620152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.620143 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e36600-22d7-49c7-8cd0-903dce78e7d3" containerName="revoke-keys" Nov 01 05:23:29.623438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.623422 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.666400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.666367 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.666545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.666521 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpc4j\" (UniqueName: \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.666648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.666627 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.682549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.682519 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v"] Nov 01 05:23:29.727091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.727064 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:23:29.727091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.727087 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:23:29.727331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.727135 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:23:29.728278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.728239 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:23:29.767389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.767358 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.767580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.767415 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.767580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.767544 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpc4j\" (UniqueName: \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.768132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.768113 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.769867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.769849 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.831029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.830988 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpc4j\" (UniqueName: \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\") pod \"maas-api-revoke-sub-89f54590-zmj4v\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:29.937315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:29.937231 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:30.316920 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:30.316889 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4b338b_5f8e_48eb_9bab_2ce1ce8f63a3.slice/crio-044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260 WatchSource:0}: Error finding container 044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260: Status 404 returned error can't find the container with id 044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260 Nov 01 05:23:30.317399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:30.317374 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v"] Nov 01 05:23:30.587759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:30.587677 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:23:30.832155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:30.832114 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" event={"ID":"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3","Type":"ContainerStarted","Data":"e55872acb3de70f45fb96b0a5e6c6abc92e60cef64fb1a659eb3d77387d32bc3"} Nov 01 05:23:30.832669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:30.832199 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" event={"ID":"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3","Type":"ContainerStarted","Data":"044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260"} Nov 01 05:23:30.954703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:30.954600 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" podStartSLOduration=1.954584697 podStartE2EDuration="1.954584697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:23:30.950330852 +0000 UTC m=+1226.090919773" watchObservedRunningTime="2026-11-01 05:23:30.954584697 +0000 UTC m=+1226.095173616" Nov 01 05:23:31.841731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:31.841696 4425 generic.go:356] "Generic (PLEG): container finished" podID="bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" containerID="e55872acb3de70f45fb96b0a5e6c6abc92e60cef64fb1a659eb3d77387d32bc3" exitCode=0 Nov 01 05:23:31.842134 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:31.841779 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" event={"ID":"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3","Type":"ContainerDied","Data":"e55872acb3de70f45fb96b0a5e6c6abc92e60cef64fb1a659eb3d77387d32bc3"} Nov 01 05:23:33.005354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.005324 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:33.049550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.049519 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4"] Nov 01 05:23:33.053667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.053644 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce61f67-jp8w4"] Nov 01 05:23:33.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.119812 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\" (UniqueName: \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\") pod \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " Nov 01 05:23:33.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.119871 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\") pod \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " Nov 01 05:23:33.120151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.120043 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\") pod \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\" (UID: \"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3\") " Nov 01 05:23:33.120291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.120235 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca" pod "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" (UID: "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:33.120569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.120548 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:33.122281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.122241 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j" pod "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" (UID: "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3"). InnerVolumeSpecName "kube-api-access-jpc4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:33.122370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.122333 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca" pod "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" (UID: "bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:33.221846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.221809 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:33.221846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.221842 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpc4j\" (UniqueName: \"kubernetes.io/projected/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3-kube-api-access-jpc4j\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:33.858809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.858777 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" Nov 01 05:23:33.858809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.858802 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v" event={"ID":"bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3","Type":"ContainerDied","Data":"044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260"} Nov 01 05:23:33.859031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:33.858858 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044699a71858eb7405959b3af4ef4b5bdde4c21263683060a3c7632cf1881260" Nov 01 05:23:34.989553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:34.989522 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98a7a0c-f4a2-49c0-be2d-f7235cd833bb" path="/var/lib/kubelet/pods/a98a7a0c-f4a2-49c0-be2d-f7235cd833bb/volumes" Nov 01 05:23:35.096589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.096563 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rfb94_ecdd9883-e2d7-48e4-b283-f6f3137e7065/payload-processing/0.log" Nov 01 05:23:35.329911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.329882 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rfb94_ecdd9883-e2d7-48e4-b283-f6f3137e7065/payload-processing/0.log" Nov 01 05:23:35.588825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.588728 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" podUID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" containerName="payload-processing" containerID="cri-o://14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3" gracePeriod=25 Nov 01 05:23:35.874036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.874007 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:23:35.875894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.875855 4425 generic.go:356] "Generic (PLEG): container finished" podID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" containerID="14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3" exitCode=0 Nov 01 05:23:35.875894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.875890 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" Nov 01 05:23:35.876111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.875938 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" event={"ID":"aa1e42d2-6a87-434c-96d9-c32438e3cdfa","Type":"ContainerDied","Data":"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3"} Nov 01 05:23:35.876111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.876002 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7" event={"ID":"aa1e42d2-6a87-434c-96d9-c32438e3cdfa","Type":"ContainerDied","Data":"b64906d4345958cde16bb4f92974be98d6fe87c40c1f15462e6dbe5103e157ec"} Nov 01 05:23:35.876111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.876034 4425 scope.go:122] "RemoveContainer" containerID="14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3" Nov 01 05:23:35.887892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.887872 4425 scope.go:122] "RemoveContainer" containerID="14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3" Nov 01 05:23:35.888207 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:23:35.888186 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3\": container with ID starting with 14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3 not found: ID does not exist" containerID="14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3" Nov 01 05:23:35.888288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.888220 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3"} err="failed to get container status \"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3\": rpc error: code = NotFound desc = could not find container \"14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3\": container with ID starting with 14294295e32ab2e2d500dd62b4c9a2a61263f7d7370c7f65794d4bc5721abfe3 not found: ID does not exist" Nov 01 05:23:35.957152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.957109 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\") pod \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " Nov 01 05:23:35.957380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.957201 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\" (UniqueName: \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\") pod \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\" (UID: \"aa1e42d2-6a87-434c-96d9-c32438e3cdfa\") " Nov 01 05:23:35.957581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.957555 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume" pod "aa1e42d2-6a87-434c-96d9-c32438e3cdfa" (UID: "aa1e42d2-6a87-434c-96d9-c32438e3cdfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:35.959591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:35.959562 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b" pod "aa1e42d2-6a87-434c-96d9-c32438e3cdfa" (UID: "aa1e42d2-6a87-434c-96d9-c32438e3cdfa"). InnerVolumeSpecName "kube-api-access-pp25b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:36.059522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:36.059456 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:36.059522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:36.059493 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp25b\" (UniqueName: \"kubernetes.io/projected/aa1e42d2-6a87-434c-96d9-c32438e3cdfa-kube-api-access-pp25b\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:36.302831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:36.302784 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:23:36.306970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:36.306949 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-84846d58f9-n87g7"] Nov 01 05:23:36.974401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:36.974367 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" path="/var/lib/kubelet/pods/aa1e42d2-6a87-434c-96d9-c32438e3cdfa/volumes" Nov 01 05:23:37.150341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.150307 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct"] Nov 01 05:23:37.150895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.150881 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" containerName="revoke-keys" Nov 01 05:23:37.150951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.150896 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" containerName="revoke-keys" Nov 01 05:23:37.150951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.150911 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" containerName="payload-processing" Nov 01 05:23:37.150951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.150917 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" containerName="payload-processing" Nov 01 05:23:37.151112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.151021 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" containerName="revoke-keys" Nov 01 05:23:37.151112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.151031 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1e42d2-6a87-434c-96d9-c32438e3cdfa" containerName="payload-processing" Nov 01 05:23:37.155959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.155937 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.212502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.212470 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:23:37.212951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.212930 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct"] Nov 01 05:23:37.258475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.258391 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:23:37.258475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.258409 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:23:37.258475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.258409 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:23:37.275699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.275663 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.275991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.275763 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtthm\" (UniqueName: \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.275991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.275823 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.376874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.376841 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.377060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.376901 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtthm\" (UniqueName: \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.377060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.376938 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.377660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.377634 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.379368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.379351 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.440436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.440405 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtthm\" (UniqueName: \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\") pod \"maas-api-revoke-sub-1657372f-wtlct\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.471304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.471241 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:37.864378 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:37.864349 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9157b2_81a4_4f9f_ae1d_4be42ec6559f.slice/crio-3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3 WatchSource:0}: Error finding container 3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3: Status 404 returned error can't find the container with id 3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3 Nov 01 05:23:37.866141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.866118 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct"] Nov 01 05:23:37.895915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:37.895881 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" event={"ID":"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f","Type":"ContainerStarted","Data":"3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3"} Nov 01 05:23:38.914526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:38.914491 4425 generic.go:356] "Generic (PLEG): container finished" podID="7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" containerID="d254099c34bbc210b985c10cbfd05931e194faa1fefd61654d8ed22198d505b8" exitCode=0 Nov 01 05:23:38.914925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:38.914583 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" event={"ID":"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f","Type":"ContainerDied","Data":"d254099c34bbc210b985c10cbfd05931e194faa1fefd61654d8ed22198d505b8"} Nov 01 05:23:40.116927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.116902 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:40.231883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.231796 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\" (UniqueName: \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\") pod \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " Nov 01 05:23:40.232031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.231979 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\") pod \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " Nov 01 05:23:40.232104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.232086 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\") pod \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\" (UID: \"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f\") " Nov 01 05:23:40.232428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.232399 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca" pod "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" (UID: "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:40.234133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.234110 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm" pod "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" (UID: "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f"). InnerVolumeSpecName "kube-api-access-qtthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:40.234281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.234240 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca" pod "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" (UID: "7c9157b2-81a4-4f9f-ae1d-4be42ec6559f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:40.333664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.333633 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:40.333664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.333666 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:40.333856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.333680 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtthm\" (UniqueName: \"kubernetes.io/projected/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f-kube-api-access-qtthm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:40.977274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.970857 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" event={"ID":"7c9157b2-81a4-4f9f-ae1d-4be42ec6559f","Type":"ContainerDied","Data":"3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3"} Nov 01 05:23:40.977274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.970927 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3346ab7016c7d456dd99ea45175d29ea9c241f90f85234295e39b7496fff56a3" Nov 01 05:23:40.977274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:40.970999 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct" Nov 01 05:23:45.050073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.050036 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq"] Nov 01 05:23:45.100233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.100199 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937c8bea-gthjq"] Nov 01 05:23:45.155680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.155650 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:23:45.155939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.155897 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" containerID="cri-o://5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465" gracePeriod=30 Nov 01 05:23:45.277196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.275810 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294"] Nov 01 05:23:45.277196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.276782 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" containerName="revoke-keys" Nov 01 05:23:45.277196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.276802 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" containerName="revoke-keys" Nov 01 05:23:45.277196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.276956 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" containerName="revoke-keys" Nov 01 05:23:45.307332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.300587 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.381880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.381844 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlhk\" (UniqueName: \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.382046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.381920 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.382112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.382079 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.382683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.382665 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294"] Nov 01 05:23:45.386041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.386014 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:23:45.386156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.386126 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:23:45.436839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.436809 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:23:45.436839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.436831 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:23:45.483358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.483311 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.483515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.483464 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlhk\" (UniqueName: \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.483515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.483506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.484101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.484081 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.486121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.486096 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.546705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.546671 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlhk\" (UniqueName: \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\") pod \"maas-api-revoke-sub-908f41d5-n8294\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:45.700879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:45.700849 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:46.089604 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:46.089570 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae36317_fff1_44e2_9097_cb863e865901.slice/crio-c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9 WatchSource:0}: Error finding container c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9: Status 404 returned error can't find the container with id c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9 Nov 01 05:23:46.092169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:46.092144 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294"] Nov 01 05:23:46.974158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:46.974120 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d60ee8b-77c2-4230-8bd9-68148ec52722" path="/var/lib/kubelet/pods/7d60ee8b-77c2-4230-8bd9-68148ec52722/volumes" Nov 01 05:23:47.066295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.066238 4425 generic.go:356] "Generic (PLEG): container finished" podID="9ae36317-fff1-44e2-9097-cb863e865901" containerID="fa0d2f2084f202f64fd9766c4278c8cd1a82b8cb358e4f6a30bfcf2f825e6efb" exitCode=0 Nov 01 05:23:47.066470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.066296 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" event={"ID":"9ae36317-fff1-44e2-9097-cb863e865901","Type":"ContainerDied","Data":"fa0d2f2084f202f64fd9766c4278c8cd1a82b8cb358e4f6a30bfcf2f825e6efb"} Nov 01 05:23:47.066470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.066357 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" event={"ID":"9ae36317-fff1-44e2-9097-cb863e865901","Type":"ContainerStarted","Data":"c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9"} Nov 01 05:23:47.896194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.896165 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w"] Nov 01 05:23:47.900471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.900453 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:47.960571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:47.960537 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w"] Nov 01 05:23:48.014192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.014152 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.014456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.014432 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkwr\" (UniqueName: \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.014563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.014550 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.115921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.115875 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.116100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.116003 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.116183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.116164 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkwr\" (UniqueName: \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.116548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.116525 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.118631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.118609 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.179704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.179630 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkwr\" (UniqueName: \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\") pod \"maas-api-revoke-sub-6819094b-rnv6w\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.217141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.217064 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:48.258030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.257997 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:48.319039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.319007 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\") pod \"9ae36317-fff1-44e2-9097-cb863e865901\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " Nov 01 05:23:48.319223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.319162 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\") pod \"9ae36317-fff1-44e2-9097-cb863e865901\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " Nov 01 05:23:48.319329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.319226 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\" (UniqueName: \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\") pod \"9ae36317-fff1-44e2-9097-cb863e865901\" (UID: \"9ae36317-fff1-44e2-9097-cb863e865901\") " Nov 01 05:23:48.319641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.319610 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca" pod "9ae36317-fff1-44e2-9097-cb863e865901" (UID: "9ae36317-fff1-44e2-9097-cb863e865901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:48.319803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.319782 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:48.321520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.321496 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca" pod "9ae36317-fff1-44e2-9097-cb863e865901" (UID: "9ae36317-fff1-44e2-9097-cb863e865901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:48.321940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.321918 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk" pod "9ae36317-fff1-44e2-9097-cb863e865901" (UID: "9ae36317-fff1-44e2-9097-cb863e865901"). InnerVolumeSpecName "kube-api-access-fwlhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:48.421101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.421064 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae36317-fff1-44e2-9097-cb863e865901-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:48.421101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.421094 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlhk\" (UniqueName: \"kubernetes.io/projected/9ae36317-fff1-44e2-9097-cb863e865901-kube-api-access-fwlhk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:48.601079 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:23:48.601044 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7166c2_4d0d_4c9d_b575_b0b9a4b75ed9.slice/crio-d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201 WatchSource:0}: Error finding container d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201: Status 404 returned error can't find the container with id d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201 Nov 01 05:23:48.606263 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:48.606201 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w"] Nov 01 05:23:49.049363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.049335 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p"] Nov 01 05:23:49.053444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.053422 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78055817-bzn5p"] Nov 01 05:23:49.085298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.085241 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" event={"ID":"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9","Type":"ContainerStarted","Data":"1413f30a01380749f6902ef33f510131a232f04c657827dd76c6b01075d34c6d"} Nov 01 05:23:49.085464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.085322 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" event={"ID":"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9","Type":"ContainerStarted","Data":"d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201"} Nov 01 05:23:49.086817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.086796 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" Nov 01 05:23:49.086945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.086830 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294" event={"ID":"9ae36317-fff1-44e2-9097-cb863e865901","Type":"ContainerDied","Data":"c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9"} Nov 01 05:23:49.086945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.086880 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c922b3c816ac3f16098ea1932bcdb3204c71a1e4b6705dd3417b3811b261a8d9" Nov 01 05:23:49.207931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:49.207822 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" podStartSLOduration=2.207808859 podStartE2EDuration="2.207808859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:23:49.203286941 +0000 UTC m=+1244.343875882" watchObservedRunningTime="2026-11-01 05:23:49.207808859 +0000 UTC m=+1244.348397778" Nov 01 05:23:50.097747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.097709 4425 generic.go:356] "Generic (PLEG): container finished" podID="7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" containerID="1413f30a01380749f6902ef33f510131a232f04c657827dd76c6b01075d34c6d" exitCode=0 Nov 01 05:23:50.098196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.097798 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" event={"ID":"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9","Type":"ContainerDied","Data":"1413f30a01380749f6902ef33f510131a232f04c657827dd76c6b01075d34c6d"} Nov 01 05:23:50.429707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.429677 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:23:50.548352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548313 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548399 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548445 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548554 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548584 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548596 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:23:50.548740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548616 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548715 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548760 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.548952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548818 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\") pod \"2276e4f5-451a-4144-90af-e4f5f1eb5748\" (UID: \"2276e4f5-451a-4144-90af-e4f5f1eb5748\") " Nov 01 05:23:50.549050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.548912 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:23:50.549202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549174 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:23:50.549464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549292 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:50.549634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549610 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:23:50.549727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549615 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.549727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549665 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2276e4f5-451a-4144-90af-e4f5f1eb5748-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.549727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549686 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.549727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.549702 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.551050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.551024 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:23:50.551050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.551033 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "kube-api-access-9h8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:50.551226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.551063 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:23:50.551226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.551140 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token" pod "2276e4f5-451a-4144-90af-e4f5f1eb5748" (UID: "2276e4f5-451a-4144-90af-e4f5f1eb5748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:50.650983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.650907 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8lj\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-kube-api-access-9h8lj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.650983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.650943 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.650983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.650957 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.650983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.650969 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.650983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.650982 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2276e4f5-451a-4144-90af-e4f5f1eb5748-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:50.974548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:50.974511 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dab1e00-deca-475c-ba24-2e0177671a59" path="/var/lib/kubelet/pods/6dab1e00-deca-475c-ba24-2e0177671a59/volumes" Nov 01 05:23:51.107693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.107658 4425 generic.go:356] "Generic (PLEG): container finished" podID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerID="5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465" exitCode=0 Nov 01 05:23:51.108071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.107721 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" Nov 01 05:23:51.108071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.107735 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerDied","Data":"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465"} Nov 01 05:23:51.108071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.107797 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6" event={"ID":"2276e4f5-451a-4144-90af-e4f5f1eb5748","Type":"ContainerDied","Data":"11116d9da5fb0519ede9c7aec7c83d5750d6a183713c07aa6d40a370e149fcc6"} Nov 01 05:23:51.108071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.107821 4425 scope.go:122] "RemoveContainer" containerID="5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465" Nov 01 05:23:51.121674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.120988 4425 scope.go:122] "RemoveContainer" containerID="9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c" Nov 01 05:23:51.134448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.134416 4425 scope.go:122] "RemoveContainer" containerID="5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465" Nov 01 05:23:51.134882 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:23:51.134858 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465\": container with ID starting with 5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465 not found: ID does not exist" containerID="5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465" Nov 01 05:23:51.134991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.134892 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465"} err="failed to get container status \"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465\": rpc error: code = NotFound desc = could not find container \"5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465\": container with ID starting with 5be83d46a8841e07837cc33928a4cc2f301f211432696495fd189a5e1e651465 not found: ID does not exist" Nov 01 05:23:51.134991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.134912 4425 scope.go:122] "RemoveContainer" containerID="9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c" Nov 01 05:23:51.135223 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:23:51.135205 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c\": container with ID starting with 9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c not found: ID does not exist" containerID="9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c" Nov 01 05:23:51.135317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.135231 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c"} err="failed to get container status \"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c\": rpc error: code = NotFound desc = could not find container \"9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c\": container with ID starting with 9d6741a1b1e4a0d129ca03e307c3ea68c9b6a809e24eb3859d02bdf8043a944c not found: ID does not exist" Nov 01 05:23:51.271478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.271453 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:51.286754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.286721 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:23:51.335070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.335036 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ea125-openshift-default-dd6f78f5f-74lb6"] Nov 01 05:23:51.357637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.357608 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\") pod \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " Nov 01 05:23:51.357773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.357696 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\") pod \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " Nov 01 05:23:51.357829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.357797 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\" (UniqueName: \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\") pod \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\" (UID: \"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9\") " Nov 01 05:23:51.358054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.358026 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca" pod "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" (UID: "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:23:51.358306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.358289 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:51.359938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.359920 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca" pod "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" (UID: "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:23:51.360101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.360074 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr" pod "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" (UID: "7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9"). InnerVolumeSpecName "kube-api-access-bzkwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:23:51.459721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.459684 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:51.459721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:51.459715 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzkwr\" (UniqueName: \"kubernetes.io/projected/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9-kube-api-access-bzkwr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:23:52.117202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:52.117172 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" Nov 01 05:23:52.117636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:52.117205 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w" event={"ID":"7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9","Type":"ContainerDied","Data":"d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201"} Nov 01 05:23:52.117636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:52.117264 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d54fe30c98c3086a5ca3dd4a3e3bd25dab39bc18a4ccb87810248234cb56a201" Nov 01 05:23:52.974336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:52.974305 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" path="/var/lib/kubelet/pods/2276e4f5-451a-4144-90af-e4f5f1eb5748/volumes" Nov 01 05:23:53.460463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:53.460427 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:23:56.949553 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:23:56.949514 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:23:58.515648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:58.515571 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" containerID="cri-o://76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7" gracePeriod=25 Nov 01 05:23:59.697851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:59.697809 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:59.760438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:59.754361 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:59.780426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:59.780379 4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:23:59.840943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:23:59.840533 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" podUID="27bad948-9ce6-42ff-b5e2-3deaa8d02069" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:01.067728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.067690 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn"] Nov 01 05:24:01.117756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.117722 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3928d7fd-8c4hn"] Nov 01 05:24:01.412823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.412780 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv"] Nov 01 05:24:01.413575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413554 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" containerName="revoke-keys" Nov 01 05:24:01.413575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413574 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" containerName="revoke-keys" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413608 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9ae36317-fff1-44e2-9097-cb863e865901" containerName="revoke-keys" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413614 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae36317-fff1-44e2-9097-cb863e865901" containerName="revoke-keys" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413630 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413635 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413652 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413656 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413743 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.413746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413751 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae36317-fff1-44e2-9097-cb863e865901" containerName="revoke-keys" Nov 01 05:24:01.414129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413757 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" containerName="revoke-keys" Nov 01 05:24:01.414129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.413768 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2276e4f5-451a-4144-90af-e4f5f1eb5748" containerName="istio-proxy" Nov 01 05:24:01.417583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.417497 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.475189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.475149 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv"] Nov 01 05:24:01.505330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.505289 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.505582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.505498 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.505582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.505563 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htzl\" (UniqueName: \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.518090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.518058 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:24:01.519215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.518239 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:24:01.519215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.518370 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:24:01.519215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.518419 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:24:01.606962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.606929 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.607174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.606981 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htzl\" (UniqueName: \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.607174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.607111 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.607872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.607786 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.609725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.609704 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.670305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.670187 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htzl\" (UniqueName: \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\") pod \"maas-api-revoke-sub-3a35340a-zmjfv\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:01.737982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:01.734471 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:02.137685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:02.137650 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv"] Nov 01 05:24:02.138339 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:02.138306 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2322d14_f4f2_4bef_b294_1a49550b582f.slice/crio-8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f WatchSource:0}: Error finding container 8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f: Status 404 returned error can't find the container with id 8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f Nov 01 05:24:02.314878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:02.314843 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" event={"ID":"c2322d14-f4f2-4bef-b294-1a49550b582f","Type":"ContainerStarted","Data":"8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f"} Nov 01 05:24:02.984401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:02.984366 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1a55c1-f296-497b-a8e0-46bd8450f606" path="/var/lib/kubelet/pods/ec1a55c1-f296-497b-a8e0-46bd8450f606/volumes" Nov 01 05:24:03.324813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:03.324780 4425 generic.go:356] "Generic (PLEG): container finished" podID="c2322d14-f4f2-4bef-b294-1a49550b582f" containerID="f42e4e648b0bd7fe7dad724cc72772aace774fa036eb36e5a8e5b824ce25d8f2" exitCode=0 Nov 01 05:24:03.325356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:03.324844 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" event={"ID":"c2322d14-f4f2-4bef-b294-1a49550b582f","Type":"ContainerDied","Data":"f42e4e648b0bd7fe7dad724cc72772aace774fa036eb36e5a8e5b824ce25d8f2"} Nov 01 05:24:04.477678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.477637 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz"] Nov 01 05:24:04.483445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.483424 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.506436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.506410 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:04.539687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.539649 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz"] Nov 01 05:24:04.597629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.597535 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\") pod \"c2322d14-f4f2-4bef-b294-1a49550b582f\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " Nov 01 05:24:04.597629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.597624 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\") pod \"c2322d14-f4f2-4bef-b294-1a49550b582f\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " Nov 01 05:24:04.597871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.597739 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\" (UniqueName: \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\") pod \"c2322d14-f4f2-4bef-b294-1a49550b582f\" (UID: \"c2322d14-f4f2-4bef-b294-1a49550b582f\") " Nov 01 05:24:04.597956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.597928 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca" pod "c2322d14-f4f2-4bef-b294-1a49550b582f" (UID: "c2322d14-f4f2-4bef-b294-1a49550b582f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:04.598065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.598049 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.598332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.598309 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.598594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.598577 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2z9\" (UniqueName: \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.598751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.598734 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:04.600097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.600071 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca" pod "c2322d14-f4f2-4bef-b294-1a49550b582f" (UID: "c2322d14-f4f2-4bef-b294-1a49550b582f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:04.600176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.600133 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl" pod "c2322d14-f4f2-4bef-b294-1a49550b582f" (UID: "c2322d14-f4f2-4bef-b294-1a49550b582f"). InnerVolumeSpecName "kube-api-access-2htzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:04.699541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.699505 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.699717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.699656 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.699717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.699704 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2z9\" (UniqueName: \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.699809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.699799 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htzl\" (UniqueName: \"kubernetes.io/projected/c2322d14-f4f2-4bef-b294-1a49550b582f-kube-api-access-2htzl\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:04.699868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.699818 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322d14-f4f2-4bef-b294-1a49550b582f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:04.700146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.700117 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.702329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.702307 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.763461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.763424 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2z9\" (UniqueName: \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\") pod \"maas-api-revoke-sub-a2eeea74-6hxvz\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.816455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.816423 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:04.911757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:04.911712 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:24:05.060875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.060836 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr"] Nov 01 05:24:05.066321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.066289 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36809939-5kgqr"] Nov 01 05:24:05.164647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.164567 4425 scope.go:122] "RemoveContainer" containerID="845d3632aaa38709ebc77d260ae241128d4441559a7ed0f0c2e8815c822ddefe" Nov 01 05:24:05.176290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.176240 4425 scope.go:122] "RemoveContainer" containerID="cc1abb4545ff23a9691397a0b39b20dd694b9c088eb455c799ad3595aec71216" Nov 01 05:24:05.196502 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:05.196475 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a76bd4_48f3_4d0e_a452_06a05cdca804.slice/crio-83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5 WatchSource:0}: Error finding container 83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5: Status 404 returned error can't find the container with id 83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5 Nov 01 05:24:05.197556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.197536 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz"] Nov 01 05:24:05.345619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.345594 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" Nov 01 05:24:05.345803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.345634 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv" event={"ID":"c2322d14-f4f2-4bef-b294-1a49550b582f","Type":"ContainerDied","Data":"8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f"} Nov 01 05:24:05.345803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.345688 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a38a6befd5be55df5a08809dfa1f34e0db0834f3b46db4dc5db97d66391c75f" Nov 01 05:24:05.347268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:05.347228 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" event={"ID":"70a76bd4-48f3-4d0e-a452-06a05cdca804","Type":"ContainerStarted","Data":"83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5"} Nov 01 05:24:06.115294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:06.115229 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:24:06.115786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:06.115528 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" containerID="cri-o://8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4" gracePeriod=30 Nov 01 05:24:06.357996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:06.357959 4425 generic.go:356] "Generic (PLEG): container finished" podID="70a76bd4-48f3-4d0e-a452-06a05cdca804" containerID="852b611a864b5f079deb69d8712a25671f70f60bb11178e67b917f8d506bf3a1" exitCode=0 Nov 01 05:24:06.358184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:06.358051 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" event={"ID":"70a76bd4-48f3-4d0e-a452-06a05cdca804","Type":"ContainerDied","Data":"852b611a864b5f079deb69d8712a25671f70f60bb11178e67b917f8d506bf3a1"} Nov 01 05:24:06.974311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:06.974272 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fecc27e-8d20-46b4-8e1d-218f3946c486" path="/var/lib/kubelet/pods/5fecc27e-8d20-46b4-8e1d-218f3946c486/volumes" Nov 01 05:24:07.507460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.507436 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:07.536116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.536083 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\" (UniqueName: \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\") pod \"70a76bd4-48f3-4d0e-a452-06a05cdca804\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " Nov 01 05:24:07.536287 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.536210 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\") pod \"70a76bd4-48f3-4d0e-a452-06a05cdca804\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " Nov 01 05:24:07.536356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.536342 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\") pod \"70a76bd4-48f3-4d0e-a452-06a05cdca804\" (UID: \"70a76bd4-48f3-4d0e-a452-06a05cdca804\") " Nov 01 05:24:07.536577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.536556 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca" pod "70a76bd4-48f3-4d0e-a452-06a05cdca804" (UID: "70a76bd4-48f3-4d0e-a452-06a05cdca804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:07.536734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.536718 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:07.538511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.538486 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca" pod "70a76bd4-48f3-4d0e-a452-06a05cdca804" (UID: "70a76bd4-48f3-4d0e-a452-06a05cdca804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:07.538605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.538579 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9" pod "70a76bd4-48f3-4d0e-a452-06a05cdca804" (UID: "70a76bd4-48f3-4d0e-a452-06a05cdca804"). InnerVolumeSpecName "kube-api-access-mb2z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:07.638452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.638372 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a76bd4-48f3-4d0e-a452-06a05cdca804-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:07.638452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:07.638400 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2z9\" (UniqueName: \"kubernetes.io/projected/70a76bd4-48f3-4d0e-a452-06a05cdca804-kube-api-access-mb2z9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:08.222320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222282 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:24:08.222893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222878 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c2322d14-f4f2-4bef-b294-1a49550b582f" containerName="revoke-keys" Nov 01 05:24:08.222945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222895 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2322d14-f4f2-4bef-b294-1a49550b582f" containerName="revoke-keys" Nov 01 05:24:08.222945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222908 4425 cpu_manager.go:414] "containerMap: removing container" podUID="70a76bd4-48f3-4d0e-a452-06a05cdca804" containerName="revoke-keys" Nov 01 05:24:08.222945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222913 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a76bd4-48f3-4d0e-a452-06a05cdca804" containerName="revoke-keys" Nov 01 05:24:08.223036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.222997 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2322d14-f4f2-4bef-b294-1a49550b582f" containerName="revoke-keys" Nov 01 05:24:08.223036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.223014 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a76bd4-48f3-4d0e-a452-06a05cdca804" containerName="revoke-keys" Nov 01 05:24:08.226750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.226734 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.286983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.286930 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-918b8d-openshift-default-dockercfg-4lb4z\"" Nov 01 05:24:08.344678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.344645 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:24:08.346076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346032 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346214 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346296 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346324 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346391 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6g5t\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346436 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346459 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346493 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.346586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.346547 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.375373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.375346 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" Nov 01 05:24:08.375742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.375384 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz" event={"ID":"70a76bd4-48f3-4d0e-a452-06a05cdca804","Type":"ContainerDied","Data":"83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5"} Nov 01 05:24:08.375742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.375443 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ac93f80dc5a82454f87d7e0e130b46285137e6955991a40b792cfbab74c5e5" Nov 01 05:24:08.448007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.447970 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448012 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448141 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448218 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448311 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448470 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448530 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448549 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448548 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448571 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448689 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6g5t\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.448842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.448789 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.449128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.449111 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.451589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.451131 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.451589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.451331 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.511604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.511525 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.511956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.511703 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6g5t\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\") pod \"e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.540913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.540883 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:08.986453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.981628 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:24:08.988927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.988891 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:08.989018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.988963 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:08.989018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:08.988996 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:09.385313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:09.385272 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerStarted","Data":"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03"} Nov 01 05:24:09.385544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:09.385339 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerStarted","Data":"0803a68a785899ef5d945e48356c18946c06a167cd3fbf3e15eec5a1c83c6621"} Nov 01 05:24:09.513824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:09.513768 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podStartSLOduration=1.5137503799999998 podStartE2EDuration="1.51375038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:09.508071828 +0000 UTC m=+1264.648660748" watchObservedRunningTime="2026-11-01 05:24:09.51375038 +0000 UTC m=+1264.654339299" Nov 01 05:24:09.541114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:09.541080 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:10.547006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:10.546977 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:11.383901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.383873 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:24:11.418072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.417978 4425 generic.go:356] "Generic (PLEG): container finished" podID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerID="8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4" exitCode=0 Nov 01 05:24:11.418072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.418043 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" Nov 01 05:24:11.418072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.418047 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerDied","Data":"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4"} Nov 01 05:24:11.418356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.418118 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt" event={"ID":"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff","Type":"ContainerDied","Data":"9a62b035d055ec4d17da2820c1ee5dc8f4060dd34d88edce1e7b8c12f6fb18fd"} Nov 01 05:24:11.418356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.418143 4425 scope.go:122] "RemoveContainer" containerID="8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4" Nov 01 05:24:11.418469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.418455 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:11.419992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.419892 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:24:11.429622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.429601 4425 scope.go:122] "RemoveContainer" containerID="cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5" Nov 01 05:24:11.440724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.440701 4425 scope.go:122] "RemoveContainer" containerID="8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4" Nov 01 05:24:11.441104 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:24:11.441084 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4\": container with ID starting with 8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4 not found: ID does not exist" containerID="8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4" Nov 01 05:24:11.441169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.441115 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4"} err="failed to get container status \"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4\": rpc error: code = NotFound desc = could not find container \"8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4\": container with ID starting with 8b0f20ad6d24fa2b9b6b8591630b32ade16b5b4c7d2fe2ccefcf0bc07e6cb9c4 not found: ID does not exist" Nov 01 05:24:11.441169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.441134 4425 scope.go:122] "RemoveContainer" containerID="cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5" Nov 01 05:24:11.441417 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:24:11.441400 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5\": container with ID starting with cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5 not found: ID does not exist" containerID="cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5" Nov 01 05:24:11.441455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.441424 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5"} err="failed to get container status \"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5\": rpc error: code = NotFound desc = could not find container \"cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5\": container with ID starting with cf945fd93b1882d19e724f694d2576b4eb6fd665a4e8aee72660f51a635973d5 not found: ID does not exist" Nov 01 05:24:11.500530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500500 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500624 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500693 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500736 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500784 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500812 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.500982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.500963 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.501060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501043 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.501119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501079 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\") pod \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\" (UID: \"8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff\") " Nov 01 05:24:11.501311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501286 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:24:11.501311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501190 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501500 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501510 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.501530 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.502010 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.502025 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.502038 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.502049 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.502101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.502065 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.503439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.503410 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:24:11.503687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.503665 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:11.503799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.503692 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "kube-api-access-vtxrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:11.504101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.504084 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy" pod "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" (UID: "8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:24:11.602822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.602790 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxrr\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-kube-api-access-vtxrr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.602822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.602817 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.602822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.602827 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.603296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.602836 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:11.882766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.882730 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:24:11.886726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:11.886704 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-af8051-openshift-default-6cb5867fc5-lk6dt"] Nov 01 05:24:13.000354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:13.000187 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" path="/var/lib/kubelet/pods/8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff/volumes" Nov 01 05:24:14.739328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:14.739284 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:14.837286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:14.837219 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" podUID="27bad948-9ce6-42ff-b5e2-3deaa8d02069" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:14.929403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:14.929359 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:24:15.462506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.462473 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx"] Nov 01 05:24:15.463071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.463058 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:15.463116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.463073 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:15.463116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.463094 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:15.463116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.463099 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:15.463213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.463194 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:15.468149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.468127 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.524693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.524661 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx"] Nov 01 05:24:15.569529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.569481 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:24:15.569529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.569503 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:24:15.569750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.569538 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:24:15.570112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.570092 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:24:15.582512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.582479 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.582659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.582613 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.582703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.582658 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pphw\" (UniqueName: \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.684562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.684524 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.684763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.684596 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pphw\" (UniqueName: \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.684763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.684754 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.685123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.685101 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.687144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.687120 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.749701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.749626 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pphw\" (UniqueName: \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\") pod \"maas-api-revoke-sub-afd0eae6-4zgfx\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:15.781903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:15.781873 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:16.186347 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:16.186319 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29d4bc_6a86_4bf7_87df_410606d60a1e.slice/crio-9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62 WatchSource:0}: Error finding container 9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62: Status 404 returned error can't find the container with id 9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62 Nov 01 05:24:16.186762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.186736 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx"] Nov 01 05:24:16.362899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.362864 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:24:16.363639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.363619 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9f4419-e6d4-41aa-8154-f24d3d8aa2ff" containerName="istio-proxy" Nov 01 05:24:16.368390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.368369 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.422226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.422193 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-918b8d\"" Nov 01 05:24:16.422458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.422232 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-918b8d-dockercfg-trjsq\"" Nov 01 05:24:16.426590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.426560 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:24:16.511208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.511158 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.511462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.511380 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxl8\" (UniqueName: \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.591344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.591300 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" event={"ID":"7e29d4bc-6a86-4bf7-87df-410606d60a1e","Type":"ContainerStarted","Data":"1dc8fcf78b713bd97b7f6f795799f5c3d180e3d5121e6335333951389d41e91b"} Nov 01 05:24:16.591667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.591631 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" event={"ID":"7e29d4bc-6a86-4bf7-87df-410606d60a1e","Type":"ContainerStarted","Data":"9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62"} Nov 01 05:24:16.613104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.613066 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:24:16.613644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.613611 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxl8\" (UniqueName: \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.613834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.613807 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.614876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.614847 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.625243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.625215 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:24:16.625398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.625359 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.677612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.677581 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxl8\" (UniqueName: \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\") pod \"payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.681441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.681413 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:16.715428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.715356 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjm6\" (UniqueName: \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.715572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.715441 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.717217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.717158 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" podStartSLOduration=1.717135326 podStartE2EDuration="1.717135326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:16.711664384 +0000 UTC m=+1271.852253304" watchObservedRunningTime="2026-11-01 05:24:16.717135326 +0000 UTC m=+1271.857724245" Nov 01 05:24:16.816893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.816859 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjm6\" (UniqueName: \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.817407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.816927 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.817772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.817745 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.879269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.879221 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjm6\" (UniqueName: \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\") pod \"payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.951788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.951754 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:16.992092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.992048 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv"] Nov 01 05:24:16.998604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:16.998582 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.057766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.056744 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f"] Nov 01 05:24:17.065166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.065107 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv"] Nov 01 05:24:17.074468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.074292 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a406644a-ng42f"] Nov 01 05:24:17.080498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.080473 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:24:17.099395 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:17.099353 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88dc102_9c16_4192_8e40_4a3f81bfce54.slice/crio-56f2ae6301d64d8d7494337a5c2b29f5a7cf5ae9c724f81eb45a4d6299df70b7 WatchSource:0}: Error finding container 56f2ae6301d64d8d7494337a5c2b29f5a7cf5ae9c724f81eb45a4d6299df70b7: Status 404 returned error can't find the container with id 56f2ae6301d64d8d7494337a5c2b29f5a7cf5ae9c724f81eb45a4d6299df70b7 Nov 01 05:24:17.121075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.120336 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.121649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.121404 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjcj9\" (UniqueName: \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.121649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.121497 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.223345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.223244 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjcj9\" (UniqueName: \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.223345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.223338 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.223537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.223433 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.224161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.224139 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.225908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.225893 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.285623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.285592 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjcj9\" (UniqueName: \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\") pod \"maas-api-revoke-sub-0dff2fe7-w59vv\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.316265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.316221 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:17.380703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.380667 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:24:17.380892 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:17.380869 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9be87d_74f5_4d9d_8e85_3c8ff4e6651d.slice/crio-0e978f5b3fd5c6d11aaa1c67a9136b5084488c1bbc7b80cb091666d5cc190ce9 WatchSource:0}: Error finding container 0e978f5b3fd5c6d11aaa1c67a9136b5084488c1bbc7b80cb091666d5cc190ce9: Status 404 returned error can't find the container with id 0e978f5b3fd5c6d11aaa1c67a9136b5084488c1bbc7b80cb091666d5cc190ce9 Nov 01 05:24:17.602754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.602716 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" event={"ID":"b88dc102-9c16-4192-8e40-4a3f81bfce54","Type":"ContainerStarted","Data":"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594"} Nov 01 05:24:17.602952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.602784 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" event={"ID":"b88dc102-9c16-4192-8e40-4a3f81bfce54","Type":"ContainerStarted","Data":"56f2ae6301d64d8d7494337a5c2b29f5a7cf5ae9c724f81eb45a4d6299df70b7"} Nov 01 05:24:17.602952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.602815 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:17.604960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.604932 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" event={"ID":"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d","Type":"ContainerStarted","Data":"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9"} Nov 01 05:24:17.605092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.604986 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" event={"ID":"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d","Type":"ContainerStarted","Data":"0e978f5b3fd5c6d11aaa1c67a9136b5084488c1bbc7b80cb091666d5cc190ce9"} Nov 01 05:24:17.605092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.605015 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:17.606869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.606847 4425 generic.go:356] "Generic (PLEG): container finished" podID="7e29d4bc-6a86-4bf7-87df-410606d60a1e" containerID="1dc8fcf78b713bd97b7f6f795799f5c3d180e3d5121e6335333951389d41e91b" exitCode=0 Nov 01 05:24:17.606959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.606938 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" event={"ID":"7e29d4bc-6a86-4bf7-87df-410606d60a1e","Type":"ContainerDied","Data":"1dc8fcf78b713bd97b7f6f795799f5c3d180e3d5121e6335333951389d41e91b"} Nov 01 05:24:17.712305 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:17.712271 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc67a65f_0c8e_46fb_b6cb_a1218d409bc4.slice/crio-d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd WatchSource:0}: Error finding container d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd: Status 404 returned error can't find the container with id d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd Nov 01 05:24:17.713501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.713475 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv"] Nov 01 05:24:17.860430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.860374 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" podStartSLOduration=1.860354012 podStartE2EDuration="1.860354012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:17.727437733 +0000 UTC m=+1272.868026651" watchObservedRunningTime="2026-11-01 05:24:17.860354012 +0000 UTC m=+1273.000942931" Nov 01 05:24:17.977398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:17.977350 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" podStartSLOduration=1.977335118 podStartE2EDuration="1.977335118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:17.974909053 +0000 UTC m=+1273.115497994" watchObservedRunningTime="2026-11-01 05:24:17.977335118 +0000 UTC m=+1273.117924036" Nov 01 05:24:18.617343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.617301 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" event={"ID":"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4","Type":"ContainerStarted","Data":"f550518ad49021ed599b257f453c3f9fce9a480275a1385f76958723e7aea0ae"} Nov 01 05:24:18.617533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.617374 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" event={"ID":"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4","Type":"ContainerStarted","Data":"d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd"} Nov 01 05:24:18.746427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.745971 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" podStartSLOduration=2.745950996 podStartE2EDuration="2.745950996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:18.739836021 +0000 UTC m=+1273.880424941" watchObservedRunningTime="2026-11-01 05:24:18.745950996 +0000 UTC m=+1273.886539916" Nov 01 05:24:18.808583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.808547 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:18.856004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.855983 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:24:18.948021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.947982 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\") pod \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " Nov 01 05:24:18.948521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948044 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\" (UniqueName: \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\") pod \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " Nov 01 05:24:18.948521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948093 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\") pod \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\" (UID: \"7e29d4bc-6a86-4bf7-87df-410606d60a1e\") " Nov 01 05:24:18.948521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948160 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\") pod \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " Nov 01 05:24:18.948521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948383 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\" (UniqueName: \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\") pod \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\" (UID: \"f5359b1a-cce7-4a07-be2d-c3a6a9801891\") " Nov 01 05:24:18.948521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948409 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca" pod "7e29d4bc-6a86-4bf7-87df-410606d60a1e" (UID: "7e29d4bc-6a86-4bf7-87df-410606d60a1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:18.948748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948599 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume" pod "f5359b1a-cce7-4a07-be2d-c3a6a9801891" (UID: "f5359b1a-cce7-4a07-be2d-c3a6a9801891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:18.948948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948901 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:18.948948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.948924 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5359b1a-cce7-4a07-be2d-c3a6a9801891-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:18.950536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.950510 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw" pod "7e29d4bc-6a86-4bf7-87df-410606d60a1e" (UID: "7e29d4bc-6a86-4bf7-87df-410606d60a1e"). InnerVolumeSpecName "kube-api-access-9pphw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:18.950627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.950580 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4" pod "f5359b1a-cce7-4a07-be2d-c3a6a9801891" (UID: "f5359b1a-cce7-4a07-be2d-c3a6a9801891"). InnerVolumeSpecName "kube-api-access-ch8v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:18.950878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.950857 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca" pod "7e29d4bc-6a86-4bf7-87df-410606d60a1e" (UID: "7e29d4bc-6a86-4bf7-87df-410606d60a1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:18.976234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:18.976204 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012ff719-bf23-4769-8331-e858420faf8d" path="/var/lib/kubelet/pods/012ff719-bf23-4769-8331-e858420faf8d/volumes" Nov 01 05:24:19.050087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.050050 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8v4\" (UniqueName: \"kubernetes.io/projected/f5359b1a-cce7-4a07-be2d-c3a6a9801891-kube-api-access-ch8v4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:19.050087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.050083 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pphw\" (UniqueName: \"kubernetes.io/projected/7e29d4bc-6a86-4bf7-87df-410606d60a1e-kube-api-access-9pphw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:19.050087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.050095 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29d4bc-6a86-4bf7-87df-410606d60a1e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:19.628065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.628025 4425 generic.go:356] "Generic (PLEG): container finished" podID="fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" containerID="f550518ad49021ed599b257f453c3f9fce9a480275a1385f76958723e7aea0ae" exitCode=0 Nov 01 05:24:19.628323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.628108 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" event={"ID":"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4","Type":"ContainerDied","Data":"f550518ad49021ed599b257f453c3f9fce9a480275a1385f76958723e7aea0ae"} Nov 01 05:24:19.629597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.629572 4425 generic.go:356] "Generic (PLEG): container finished" podID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerID="76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7" exitCode=0 Nov 01 05:24:19.629715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.629626 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" Nov 01 05:24:19.629715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.629637 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" event={"ID":"f5359b1a-cce7-4a07-be2d-c3a6a9801891","Type":"ContainerDied","Data":"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7"} Nov 01 05:24:19.629715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.629671 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz" event={"ID":"f5359b1a-cce7-4a07-be2d-c3a6a9801891","Type":"ContainerDied","Data":"7effcdd1b9213d19b0f2fbc257c0b8fa4df70e9afe90df01e962d7d02c92d0f8"} Nov 01 05:24:19.629715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.629697 4425 scope.go:122] "RemoveContainer" containerID="76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7" Nov 01 05:24:19.631549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.631423 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" Nov 01 05:24:19.631549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.631421 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx" event={"ID":"7e29d4bc-6a86-4bf7-87df-410606d60a1e","Type":"ContainerDied","Data":"9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62"} Nov 01 05:24:19.631549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.631544 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d562223f8ee61be3ea332c3f8d4dfd374cdcb4d560a3e9d82ede7fe7f46cf62" Nov 01 05:24:19.640673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.640655 4425 scope.go:122] "RemoveContainer" containerID="76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7" Nov 01 05:24:19.640965 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:24:19.640945 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7\": container with ID starting with 76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7 not found: ID does not exist" containerID="76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7" Nov 01 05:24:19.641021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.640976 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7"} err="failed to get container status \"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7\": rpc error: code = NotFound desc = could not find container \"76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7\": container with ID starting with 76ae2a40a1c9c6c6b2462e3664060b74cab08b276af364cce56fca34d4c0acb7 not found: ID does not exist" Nov 01 05:24:19.865057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.865028 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:24:19.869193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:19.869166 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8051-79d78db899-chrvz"] Nov 01 05:24:20.809969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.809943 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:20.973131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.973053 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\") pod \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " Nov 01 05:24:20.973131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.973100 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\" (UniqueName: \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\") pod \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " Nov 01 05:24:20.973399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.973185 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\") pod \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\" (UID: \"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4\") " Nov 01 05:24:20.973451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.973426 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca" pod "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" (UID: "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:20.973679 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.973659 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:20.974640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.974610 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" path="/var/lib/kubelet/pods/f5359b1a-cce7-4a07-be2d-c3a6a9801891/volumes" Nov 01 05:24:20.975499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.975472 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9" pod "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" (UID: "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4"). InnerVolumeSpecName "kube-api-access-rjcj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:20.975599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:20.975515 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca" pod "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" (UID: "fc67a65f-0c8e-46fb-b6cb-a1218d409bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:21.075406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:21.075367 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjcj9\" (UniqueName: \"kubernetes.io/projected/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-kube-api-access-rjcj9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:21.075406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:21.075406 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:21.648984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:21.648952 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" Nov 01 05:24:21.648984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:21.648977 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv" event={"ID":"fc67a65f-0c8e-46fb-b6cb-a1218d409bc4","Type":"ContainerDied","Data":"d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd"} Nov 01 05:24:21.649215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:21.649026 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57db57d3700740896e22a6801020774babf694a9adcccadbb2bccc9cb4ff2bd" Nov 01 05:24:23.048561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:23.048527 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv"] Nov 01 05:24:23.101914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:23.101880 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e4a00a-fgdcv"] Nov 01 05:24:25.032228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:25.032185 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3244671b-5d2e-4ed1-a34c-4cb0e40f6734" path="/var/lib/kubelet/pods/3244671b-5d2e-4ed1-a34c-4cb0e40f6734/volumes" Nov 01 05:24:25.894282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:25.894044 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" podUID="27bad948-9ce6-42ff-b5e2-3deaa8d02069" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:26.238340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:26.237460 4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:26.948035 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:24:26.947997 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:24:28.620859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:28.620829 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:24:28.621332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:28.620915 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:24:31.715556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.715524 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m"] Nov 01 05:24:31.716131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716118 4425 cpu_manager.go:414] "containerMap: removing container" podUID="fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" containerName="revoke-keys" Nov 01 05:24:31.716181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716132 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" containerName="revoke-keys" Nov 01 05:24:31.716181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716148 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7e29d4bc-6a86-4bf7-87df-410606d60a1e" containerName="revoke-keys" Nov 01 05:24:31.716181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716154 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29d4bc-6a86-4bf7-87df-410606d60a1e" containerName="revoke-keys" Nov 01 05:24:31.716181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716174 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" Nov 01 05:24:31.716181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716180 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" Nov 01 05:24:31.716358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716318 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5359b1a-cce7-4a07-be2d-c3a6a9801891" containerName="payload-pre-processing" Nov 01 05:24:31.716358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716328 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29d4bc-6a86-4bf7-87df-410606d60a1e" containerName="revoke-keys" Nov 01 05:24:31.716358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.716336 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" containerName="revoke-keys" Nov 01 05:24:31.720123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.720108 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.780173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.780138 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m"] Nov 01 05:24:31.787060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.787021 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.787207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.787101 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgg68\" (UniqueName: \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.787367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.787342 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.825380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.825345 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:24:31.825380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.825363 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:24:31.825590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.825386 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:24:31.825590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.825421 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:24:31.888512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.888479 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.888687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.888544 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgg68\" (UniqueName: \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.888687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.888660 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.889318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.889290 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.891172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.891142 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:31.951176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:31.951143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgg68\" (UniqueName: \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\") pod \"maas-api-revoke-sub-4ba67922-nl84m\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:32.033628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:32.033549 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:32.413841 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:32.413807 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26428b27_6cfa_4acf_8b44_7dbb84c0e616.slice/crio-ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8 WatchSource:0}: Error finding container ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8: Status 404 returned error can't find the container with id ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8 Nov 01 05:24:32.414555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:32.414532 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m"] Nov 01 05:24:32.902756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:32.902719 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" event={"ID":"26428b27-6cfa-4acf-8b44-7dbb84c0e616","Type":"ContainerStarted","Data":"44bd89f4ff0712d6ee3396edf0f2cbf619979302a1ba7790474ba0478af5b04a"} Nov 01 05:24:32.903246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:32.902786 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" event={"ID":"26428b27-6cfa-4acf-8b44-7dbb84c0e616","Type":"ContainerStarted","Data":"ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8"} Nov 01 05:24:33.021866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:33.021770 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" podStartSLOduration=2.021755782 podStartE2EDuration="2.021755782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:33.020602015 +0000 UTC m=+1288.161190955" watchObservedRunningTime="2026-11-01 05:24:33.021755782 +0000 UTC m=+1288.162344747" Nov 01 05:24:33.919908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:33.919869 4425 generic.go:356] "Generic (PLEG): container finished" podID="26428b27-6cfa-4acf-8b44-7dbb84c0e616" containerID="44bd89f4ff0712d6ee3396edf0f2cbf619979302a1ba7790474ba0478af5b04a" exitCode=0 Nov 01 05:24:33.920605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:33.920042 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" event={"ID":"26428b27-6cfa-4acf-8b44-7dbb84c0e616","Type":"ContainerDied","Data":"44bd89f4ff0712d6ee3396edf0f2cbf619979302a1ba7790474ba0478af5b04a"} Nov 01 05:24:34.094587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:34.094553 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz"] Nov 01 05:24:34.099084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:34.099057 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e18c901-rsqwz"] Nov 01 05:24:34.974847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:34.974813 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="637bbb5f-caf3-4736-aaa8-5f91aff1c87b" path="/var/lib/kubelet/pods/637bbb5f-caf3-4736-aaa8-5f91aff1c87b/volumes" Nov 01 05:24:35.085176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.085152 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:35.130661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.130636 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\" (UniqueName: \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\") pod \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " Nov 01 05:24:35.130822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.130671 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\") pod \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " Nov 01 05:24:35.130822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.130793 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\") pod \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\" (UID: \"26428b27-6cfa-4acf-8b44-7dbb84c0e616\") " Nov 01 05:24:35.131018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.130998 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca" pod "26428b27-6cfa-4acf-8b44-7dbb84c0e616" (UID: "26428b27-6cfa-4acf-8b44-7dbb84c0e616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:35.131360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.131341 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:35.132863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.132838 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68" pod "26428b27-6cfa-4acf-8b44-7dbb84c0e616" (UID: "26428b27-6cfa-4acf-8b44-7dbb84c0e616"). InnerVolumeSpecName "kube-api-access-hgg68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:35.133085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.133064 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca" pod "26428b27-6cfa-4acf-8b44-7dbb84c0e616" (UID: "26428b27-6cfa-4acf-8b44-7dbb84c0e616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:35.232791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.232709 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgg68\" (UniqueName: \"kubernetes.io/projected/26428b27-6cfa-4acf-8b44-7dbb84c0e616-kube-api-access-hgg68\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:35.232791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.232742 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26428b27-6cfa-4acf-8b44-7dbb84c0e616-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:35.938075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.938044 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" Nov 01 05:24:35.938300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.938075 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m" event={"ID":"26428b27-6cfa-4acf-8b44-7dbb84c0e616","Type":"ContainerDied","Data":"ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8"} Nov 01 05:24:35.938300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:35.938165 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac966d5a0004090d4828624ae3798039ebdeb9bfa301095cded830a0f5ccccb8" Nov 01 05:24:37.050714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:37.050681 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk"] Nov 01 05:24:37.101617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:37.101587 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9be62ec-6b8fk"] Nov 01 05:24:38.974179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:38.974143 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c968b76-9af8-4cc8-9d31-2e87dfad9e1d" path="/var/lib/kubelet/pods/0c968b76-9af8-4cc8-9d31-2e87dfad9e1d/volumes" Nov 01 05:24:42.534278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.530881 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh"] Nov 01 05:24:42.534278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.531904 4425 cpu_manager.go:414] "containerMap: removing container" podUID="26428b27-6cfa-4acf-8b44-7dbb84c0e616" containerName="revoke-keys" Nov 01 05:24:42.534278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.531925 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26428b27-6cfa-4acf-8b44-7dbb84c0e616" containerName="revoke-keys" Nov 01 05:24:42.534278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.532132 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="26428b27-6cfa-4acf-8b44-7dbb84c0e616" containerName="revoke-keys" Nov 01 05:24:42.537952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.537926 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.593140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.593107 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:24:42.660910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.660881 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:24:42.686885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.665468 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:24:42.687104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.665641 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:24:42.699972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.699938 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.772322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.709375 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh"] Nov 01 05:24:42.843383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.711390 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.869282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.850435 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44zv\" (UniqueName: \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.999281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.995372 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.999281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.995464 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44zv\" (UniqueName: \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.999281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.995607 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:42.999281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:42.996544 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:43.083733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:43.078424 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:43.151422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:43.151337 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44zv\" (UniqueName: \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\") pod \"maas-api-revoke-sub-9170e97f-6qdvh\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:43.181514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:43.181471 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:43.832882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:43.832837 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 05:24:44.140900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:44.140866 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh"] Nov 01 05:24:44.210079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:44.210030 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" event={"ID":"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6","Type":"ContainerStarted","Data":"38b8db43718b4c38cbad7389daa8292c32b96313c82b805ec45c210c259c3e70"} Nov 01 05:24:45.224196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:45.224154 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" event={"ID":"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6","Type":"ContainerStarted","Data":"63f1366581f52b85f0d76ce8fa8f12452c8d901bd587fe93b2a9445ad0ca2e7e"} Nov 01 05:24:45.348691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:45.348634 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" podStartSLOduration=3.348618909 podStartE2EDuration="3.348618909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:45.344173593 +0000 UTC m=+1300.484762515" watchObservedRunningTime="2026-11-01 05:24:45.348618909 +0000 UTC m=+1300.489207828" Nov 01 05:24:46.233446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:46.233411 4425 generic.go:356] "Generic (PLEG): container finished" podID="36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" containerID="63f1366581f52b85f0d76ce8fa8f12452c8d901bd587fe93b2a9445ad0ca2e7e" exitCode=0 Nov 01 05:24:46.233849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:46.233505 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" event={"ID":"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6","Type":"ContainerDied","Data":"63f1366581f52b85f0d76ce8fa8f12452c8d901bd587fe93b2a9445ad0ca2e7e"} Nov 01 05:24:47.411871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.411838 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:47.519585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.519490 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\" (UniqueName: \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\") pod \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " Nov 01 05:24:47.519756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.519659 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\") pod \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " Nov 01 05:24:47.519815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.519794 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\") pod \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\" (UID: \"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6\") " Nov 01 05:24:47.520085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.520049 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca" pod "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" (UID: "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:24:47.520458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.520434 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:47.521858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.521830 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv" pod "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" (UID: "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6"). InnerVolumeSpecName "kube-api-access-c44zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:24:47.522084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.522061 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca" pod "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" (UID: "36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:24:47.621879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.621847 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c44zv\" (UniqueName: \"kubernetes.io/projected/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-kube-api-access-c44zv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:47.621879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:47.621876 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:24:48.057513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.057476 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh"] Nov 01 05:24:48.064066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.064040 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3910eaf-sqkvh"] Nov 01 05:24:48.250746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.250710 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" Nov 01 05:24:48.250746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.250739 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh" event={"ID":"36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6","Type":"ContainerDied","Data":"38b8db43718b4c38cbad7389daa8292c32b96313c82b805ec45c210c259c3e70"} Nov 01 05:24:48.250954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.250794 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38b8db43718b4c38cbad7389daa8292c32b96313c82b805ec45c210c259c3e70" Nov 01 05:24:48.974404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:48.974370 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beda527b-a7c9-4ee0-9add-1bad810a2c75" path="/var/lib/kubelet/pods/beda527b-a7c9-4ee0-9add-1bad810a2c75/volumes" Nov 01 05:24:50.054274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:50.053985 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58"] Nov 01 05:24:50.092273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:50.091740 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab32cbec-48h58"] Nov 01 05:24:50.975211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:50.975179 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f700d3cf-b14f-4a0d-ba51-7072244c0873" path="/var/lib/kubelet/pods/f700d3cf-b14f-4a0d-ba51-7072244c0873/volumes" Nov 01 05:24:56.961809 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:24:56.961770 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:24:57.328644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.328613 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 05:24:57.329302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.329283 4425 cpu_manager.go:414] "containerMap: removing container" podUID="36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" containerName="revoke-keys" Nov 01 05:24:57.329424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.329303 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" containerName="revoke-keys" Nov 01 05:24:57.329495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.329480 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" containerName="revoke-keys" Nov 01 05:24:57.332787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.332768 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.457590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.457532 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-dee6aa-openshift-default-dockercfg-7g2s5\"" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463039 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvvp\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463195 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463439 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463518 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463597 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463699 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.465330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.463879 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.469895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.469859 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.470045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.469950 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.473300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.473269 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 05:24:57.571954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.571913 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.571968 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572017 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572062 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvvp\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572121 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572234 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572299 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572340 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572369 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.572805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.572783 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.573052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.573004 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.573405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.573386 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.573510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.573424 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.573965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.573944 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.575061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.575039 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.576041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.576023 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.636459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.636391 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.636741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.636717 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvvp\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\") pod \"e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:57.647357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:57.647331 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:58.056753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.056675 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76"] Nov 01 05:24:58.060735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.060713 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1a15be-jpt76"] Nov 01 05:24:58.089496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.089464 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 05:24:58.104875 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:24:58.104848 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767dedf1_1554_4fb3_a527_f4c432e5fc3d.slice/crio-fe03aeabd16097124487c3dba6c3dea544d09c62ebcc017cde0d513df08bd60a WatchSource:0}: Error finding container fe03aeabd16097124487c3dba6c3dea544d09c62ebcc017cde0d513df08bd60a: Status 404 returned error can't find the container with id fe03aeabd16097124487c3dba6c3dea544d09c62ebcc017cde0d513df08bd60a Nov 01 05:24:58.107220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.107178 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:58.107358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.107307 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:58.107431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.107388 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:24:58.330753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.330721 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerStarted","Data":"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a"} Nov 01 05:24:58.330935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.330777 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerStarted","Data":"fe03aeabd16097124487c3dba6c3dea544d09c62ebcc017cde0d513df08bd60a"} Nov 01 05:24:58.456665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.456605 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podStartSLOduration=1.456592044 podStartE2EDuration="1.456592044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:58.454661338 +0000 UTC m=+1313.595250285" watchObservedRunningTime="2026-11-01 05:24:58.456592044 +0000 UTC m=+1313.597180962" Nov 01 05:24:58.647881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.647804 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:24:58.975591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:58.975507 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c953b90-60f1-476a-a601-d3faa74df00e" path="/var/lib/kubelet/pods/1c953b90-60f1-476a-a601-d3faa74df00e/volumes" Nov 01 05:24:59.653361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:24:59.653332 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:25:00.348384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.348177 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:25:00.349369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.349103 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:25:00.923773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.923734 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:25:00.931385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.931354 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:00.992401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.992361 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:00.992742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:00.992717 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015172 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015313 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015390 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864rm\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015412 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015466 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015494 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.015722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.015515 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123441 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123532 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123588 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123638 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123743 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123838 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864rm\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123885 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123956 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.123997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.124516 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.125982 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.126320 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.126606 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.127864 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.128715 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.132927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.131929 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr"] Nov 01 05:25:01.138129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.138097 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.186448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.186344 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5861eed-qxqtr"] Nov 01 05:25:01.235781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.216643 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:25:01.247190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.247119 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864rm\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.249271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.249229 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-r5dmv\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.318158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.318108 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:01.754129 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:01.754099 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718726c9_f7d8_4cd2_b064_0d327136ff1e.slice/crio-2c1a61a995a3e3ed7378b064d32f3e5f6e340da96576fa2a74faaf1373bb7ef9 WatchSource:0}: Error finding container 2c1a61a995a3e3ed7378b064d32f3e5f6e340da96576fa2a74faaf1373bb7ef9: Status 404 returned error can't find the container with id 2c1a61a995a3e3ed7378b064d32f3e5f6e340da96576fa2a74faaf1373bb7ef9 Nov 01 05:25:01.755201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.755177 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:25:01.756689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.756655 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:01.756772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.756720 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:01.756772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:01.756749 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:02.374932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:02.374899 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" event={"ID":"718726c9-f7d8-4cd2-b064-0d327136ff1e","Type":"ContainerStarted","Data":"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334"} Nov 01 05:25:02.375440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:02.374960 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" event={"ID":"718726c9-f7d8-4cd2-b064-0d327136ff1e","Type":"ContainerStarted","Data":"2c1a61a995a3e3ed7378b064d32f3e5f6e340da96576fa2a74faaf1373bb7ef9"} Nov 01 05:25:02.570881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:02.570820 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podStartSLOduration=2.570800179 podStartE2EDuration="2.570800179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:02.558639814 +0000 UTC m=+1317.699228768" watchObservedRunningTime="2026-11-01 05:25:02.570800179 +0000 UTC m=+1317.711389169" Nov 01 05:25:02.974821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:02.974785 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5cedc8-5b35-467f-842f-23851e29a91b" path="/var/lib/kubelet/pods/1f5cedc8-5b35-467f-842f-23851e29a91b/volumes" Nov 01 05:25:03.318316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:03.318286 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:03.319768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:03.319743 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:03.319872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:03.319790 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:04.318784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:04.318749 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:04.319186 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:04.318800 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:05.238558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.238525 4425 scope.go:122] "RemoveContainer" containerID="98f6e315a427c514afe310391217337abf836e8597c76e14f4a3d56a6ea97985" Nov 01 05:25:05.274424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.274397 4425 scope.go:122] "RemoveContainer" containerID="853e9454849c98a47704721afc4be8aa67d6b96b7f0eba05adfe43e064434c24" Nov 01 05:25:05.287158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.287124 4425 scope.go:122] "RemoveContainer" containerID="a002517631f52231cbec1958e6a8009211237f86df3c75a5301544f4f13b5c74" Nov 01 05:25:05.320184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.319837 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:05.320184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.319904 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:05.327224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.327194 4425 scope.go:122] "RemoveContainer" containerID="14f51f0ab2ec66d588cbfc0616adfc0155d7cd557b2e0a7b9dbaa7a171bd455b" Nov 01 05:25:05.396153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.396126 4425 scope.go:122] "RemoveContainer" containerID="64f27810e3d64c325370757f4e098f28a604da6481406ac7833215c4e9ef9704" Nov 01 05:25:05.408637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.408617 4425 scope.go:122] "RemoveContainer" containerID="ec811db705aae3a17391e9496040bf09565abc6a862eeb739b151ada410a6e90" Nov 01 05:25:05.418800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.418783 4425 scope.go:122] "RemoveContainer" containerID="7e11b82e63e23172a64ce4d076a7c4c911b1ea2a09b6393ee8271322e2cc7ea7" Nov 01 05:25:05.604342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.604314 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 05:25:05.611919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.611898 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.671286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.671236 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-dee6aa-dockercfg-ww8n9\"" Nov 01 05:25:05.671493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.671352 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-dee6aa\"" Nov 01 05:25:05.674853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.674815 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 05:25:05.717665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.717625 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.717850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.717688 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jll\" (UniqueName: \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.808747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.808663 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 05:25:05.814445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.814418 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:05.822293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.821213 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.822293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.821283 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jll\" (UniqueName: \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.822293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.822199 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 05:25:05.822537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.822506 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.884968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.884923 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jll\" (UniqueName: \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\") pod \"payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:05.922777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.922745 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4m5l\" (UniqueName: \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:05.922965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.922946 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:05.927062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:05.927037 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:06.024641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.024568 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4m5l\" (UniqueName: \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:06.024813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.024756 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:06.025954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.025931 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:06.094636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.091669 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4m5l\" (UniqueName: \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\") pod \"payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:06.138327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.138065 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:06.319356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.319326 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:06.319499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.319374 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:06.404717 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:06.404683 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0047c5_ad1b_4cbc_ba30_3d0d8b8b01f1.slice/crio-103c5e769ddd28a07b3bb7275a3e77eace83eed15f7f04558f609f1c796e3b5b WatchSource:0}: Error finding container 103c5e769ddd28a07b3bb7275a3e77eace83eed15f7f04558f609f1c796e3b5b: Status 404 returned error can't find the container with id 103c5e769ddd28a07b3bb7275a3e77eace83eed15f7f04558f609f1c796e3b5b Nov 01 05:25:06.405530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.405506 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 05:25:06.571621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:06.571591 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 05:25:06.572172 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:06.572142 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfabf84_d74e_41c4_bc9a_0c09e88c512b.slice/crio-49cfed330ae59b2565c34a34d1b17b7fedb1f4dd04067fbf9518ecb32587383b WatchSource:0}: Error finding container 49cfed330ae59b2565c34a34d1b17b7fedb1f4dd04067fbf9518ecb32587383b: Status 404 returned error can't find the container with id 49cfed330ae59b2565c34a34d1b17b7fedb1f4dd04067fbf9518ecb32587383b Nov 01 05:25:07.319099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.319062 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:07.319302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.319121 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:07.371494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.371460 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt"] Nov 01 05:25:07.380601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.380569 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.432868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.432811 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" event={"ID":"ddfabf84-d74e-41c4-bc9a-0c09e88c512b","Type":"ContainerStarted","Data":"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf"} Nov 01 05:25:07.433503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.432907 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" event={"ID":"ddfabf84-d74e-41c4-bc9a-0c09e88c512b","Type":"ContainerStarted","Data":"49cfed330ae59b2565c34a34d1b17b7fedb1f4dd04067fbf9518ecb32587383b"} Nov 01 05:25:07.433656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.433631 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt"] Nov 01 05:25:07.433961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.433925 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:07.434657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.434619 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:25:07.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.437193 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" event={"ID":"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1","Type":"ContainerStarted","Data":"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e"} Nov 01 05:25:07.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.437266 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" event={"ID":"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1","Type":"ContainerStarted","Data":"103c5e769ddd28a07b3bb7275a3e77eace83eed15f7f04558f609f1c796e3b5b"} Nov 01 05:25:07.440278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.438069 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:07.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.455127 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.455266 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.455305 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrc5\" (UniqueName: \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.475735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.475699 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:25:07.475937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.475885 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:25:07.476006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.475952 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:25:07.567215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.567161 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.567465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.567224 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrc5\" (UniqueName: \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.567465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.567444 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.569029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.568999 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.570672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.570615 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.630589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.630556 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrc5\" (UniqueName: \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\") pod \"maas-api-revoke-sub-7bbd670d-p2zbt\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.670528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.670466 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" podStartSLOduration=2.670450969 podStartE2EDuration="2.670450969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:07.667795066 +0000 UTC m=+1322.808383986" watchObservedRunningTime="2026-11-01 05:25:07.670450969 +0000 UTC m=+1322.811039917" Nov 01 05:25:07.696432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.696402 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:07.791855 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:07.791427 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" podStartSLOduration=2.791409004 podStartE2EDuration="2.791409004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:07.79121149 +0000 UTC m=+1322.931800421" watchObservedRunningTime="2026-11-01 05:25:07.791409004 +0000 UTC m=+1322.931997923" Nov 01 05:25:08.115792 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:08.115766 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d219a5_ab5f_4838_8526_6db70115ee2e.slice/crio-da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1 WatchSource:0}: Error finding container da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1: Status 404 returned error can't find the container with id da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1 Nov 01 05:25:08.116076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:08.116053 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt"] Nov 01 05:25:08.319336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:08.319301 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:08.319506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:08.319353 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:08.447471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:08.447427 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" event={"ID":"20d219a5-ab5f-4838-8526-6db70115ee2e","Type":"ContainerStarted","Data":"da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1"} Nov 01 05:25:09.318917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:09.318880 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-r5dmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 01 05:25:09.319105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:09.318949 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 01 05:25:09.456705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:09.456613 4425 generic.go:356] "Generic (PLEG): container finished" podID="20d219a5-ab5f-4838-8526-6db70115ee2e" containerID="466ba23dba92653ba4aabbf9bd3a35e6222cd6fd52a2f93c5b60591879da0680" exitCode=0 Nov 01 05:25:09.457099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:09.456697 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" event={"ID":"20d219a5-ab5f-4838-8526-6db70115ee2e","Type":"ContainerDied","Data":"466ba23dba92653ba4aabbf9bd3a35e6222cd6fd52a2f93c5b60591879da0680"} Nov 01 05:25:10.372669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.372642 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:10.373184 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.373167 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:10.374464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.374443 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:25:10.649440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.649415 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:10.810836 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.810802 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\") pod \"20d219a5-ab5f-4838-8526-6db70115ee2e\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " Nov 01 05:25:10.811010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.810897 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\" (UniqueName: \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\") pod \"20d219a5-ab5f-4838-8526-6db70115ee2e\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " Nov 01 05:25:10.811010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.810950 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\") pod \"20d219a5-ab5f-4838-8526-6db70115ee2e\" (UID: \"20d219a5-ab5f-4838-8526-6db70115ee2e\") " Nov 01 05:25:10.811450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.811422 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca" pod "20d219a5-ab5f-4838-8526-6db70115ee2e" (UID: "20d219a5-ab5f-4838-8526-6db70115ee2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:25:10.811661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.811643 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:10.813268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.813223 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5" pod "20d219a5-ab5f-4838-8526-6db70115ee2e" (UID: "20d219a5-ab5f-4838-8526-6db70115ee2e"). InnerVolumeSpecName "kube-api-access-dcrc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:25:10.813511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.813492 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca" pod "20d219a5-ab5f-4838-8526-6db70115ee2e" (UID: "20d219a5-ab5f-4838-8526-6db70115ee2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:25:10.913206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.913118 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d219a5-ab5f-4838-8526-6db70115ee2e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:10.913206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:10.913151 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrc5\" (UniqueName: \"kubernetes.io/projected/20d219a5-ab5f-4838-8526-6db70115ee2e-kube-api-access-dcrc5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:11.473559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:11.473530 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" Nov 01 05:25:11.473713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:11.473575 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt" event={"ID":"20d219a5-ab5f-4838-8526-6db70115ee2e","Type":"ContainerDied","Data":"da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1"} Nov 01 05:25:11.473713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:11.473629 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da11ae82ef11746427886280c10b6987b14cd12da5127e0cbb7693eb7a2132e1" Nov 01 05:25:14.058801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:14.058768 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr"] Nov 01 05:25:14.109372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:14.109343 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43004b85-zdsgr"] Nov 01 05:25:14.974585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:14.974550 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9225c985-39b2-4c44-b73f-858c14fda99d" path="/var/lib/kubelet/pods/9225c985-39b2-4c44-b73f-858c14fda99d/volumes" Nov 01 05:25:17.053910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:17.053867 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb"] Nov 01 05:25:17.106854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:17.106807 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8a126a-m58bb"] Nov 01 05:25:18.975517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:18.975482 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816d4584-91b0-4f65-8902-30f6acdbd18c" path="/var/lib/kubelet/pods/816d4584-91b0-4f65-8902-30f6acdbd18c/volumes" Nov 01 05:25:19.465451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:19.465419 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 05:25:19.467127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:19.467101 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 05:25:21.882138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:21.882100 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:25:21.882691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:21.882430 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" containerID="cri-o://3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8" gracePeriod=30 Nov 01 05:25:26.084536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:26.084506 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rfb94_ecdd9883-e2d7-48e4-b283-f6f3137e7065/payload-processing/0.log" Nov 01 05:25:26.316758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:26.316731 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rfb94_ecdd9883-e2d7-48e4-b283-f6f3137e7065/payload-processing/0.log" Nov 01 05:25:26.950554 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:26.950517 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:25:27.072414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.071650 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:44380->10.133.0.111:15021: read: connection reset by peer" start-of-body= Nov 01 05:25:27.072414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.071734 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:44380->10.133.0.111:15021: read: connection reset by peer" Nov 01 05:25:27.179554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.179530 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:25:27.276019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.275946 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.275992 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276069 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276106 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276154 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276196 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276278 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:25:27.276425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276352 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:25:27.276425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276392 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276444 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276457 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:25:27.276564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276510 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\") pod \"593c1831-d654-412a-b1bf-e125ed51154c\" (UID: \"593c1831-d654-412a-b1bf-e125ed51154c\") " Nov 01 05:25:27.276564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276553 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:25:27.276757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.276614 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:25:27.277082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.277025 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.277082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.277050 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.277082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.277065 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/593c1831-d654-412a-b1bf-e125ed51154c-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.277082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.277078 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.277082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.277091 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.278698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.278676 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:25:27.278791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.278769 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "kube-api-access-gkmxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:25:27.279008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.278989 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:25:27.279072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.279023 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token" pod "593c1831-d654-412a-b1bf-e125ed51154c" (UID: "593c1831-d654-412a-b1bf-e125ed51154c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:25:27.378487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.378457 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.378487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.378486 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmxk\" (UniqueName: \"kubernetes.io/projected/593c1831-d654-412a-b1bf-e125ed51154c-kube-api-access-gkmxk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.378487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.378497 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/593c1831-d654-412a-b1bf-e125ed51154c-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.378728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.378506 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/593c1831-d654-412a-b1bf-e125ed51154c-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:27.634224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.634190 4425 generic.go:356] "Generic (PLEG): container finished" podID="593c1831-d654-412a-b1bf-e125ed51154c" containerID="3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8" exitCode=0 Nov 01 05:25:27.634455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.634241 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" Nov 01 05:25:27.634455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.634276 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" event={"ID":"593c1831-d654-412a-b1bf-e125ed51154c","Type":"ContainerDied","Data":"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8"} Nov 01 05:25:27.634455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.634341 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g" event={"ID":"593c1831-d654-412a-b1bf-e125ed51154c","Type":"ContainerDied","Data":"52627ac1dfb4e41c41e0f40c28253304b7de5617fb34be465add2830c2361af8"} Nov 01 05:25:27.634455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.634365 4425 scope.go:122] "RemoveContainer" containerID="3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8" Nov 01 05:25:27.636844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.636821 4425 generic.go:356] "Generic (PLEG): container finished" podID="975f14fd-353f-4421-94f0-44f8552ddee5" containerID="31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03" exitCode=0 Nov 01 05:25:27.636957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.636876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerDied","Data":"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03"} Nov 01 05:25:27.637311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.637296 4425 scope.go:122] "RemoveContainer" containerID="31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03" Nov 01 05:25:27.639157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.639117 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:27.639247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.639216 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:27.639322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.639306 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:27.646990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.646966 4425 scope.go:122] "RemoveContainer" containerID="3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8" Nov 01 05:25:27.647291 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:27.647267 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8\": container with ID starting with 3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8 not found: ID does not exist" containerID="3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8" Nov 01 05:25:27.647366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.647289 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8"} err="failed to get container status \"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8\": rpc error: code = NotFound desc = could not find container \"3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8\": container with ID starting with 3167933e9f1ecbc8732fe79f89edd6b3b7c5719229b92cd4fed77828f5279dc8 not found: ID does not exist" Nov 01 05:25:27.932144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.932063 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:25:27.936190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:27.936170 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-4r82g"] Nov 01 05:25:28.669729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:28.669686 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerStarted","Data":"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad"} Nov 01 05:25:29.023991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.023900 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593c1831-d654-412a-b1bf-e125ed51154c" path="/var/lib/kubelet/pods/593c1831-d654-412a-b1bf-e125ed51154c/volumes" Nov 01 05:25:29.064004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.063960 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf"] Nov 01 05:25:29.077367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.077324 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30176787-twnmf"] Nov 01 05:25:29.549129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.548689 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:25:29.549129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.548811 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 01 05:25:29.549129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:29.548856 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 01 05:25:30.542301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:30.541691 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 01 05:25:30.542301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:30.541761 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 01 05:25:30.557284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:30.554367 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 05:25:30.742604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:30.740964 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:25:30.841280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:30.840351 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz" podUID="27bad948-9ce6-42ff-b5e2-3deaa8d02069" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:25:31.011563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:31.011527 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed46963-d5f2-4d87-be6a-290657ef88da" path="/var/lib/kubelet/pods/3ed46963-d5f2-4d87-be6a-290657ef88da/volumes" Nov 01 05:25:31.541246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:31.541214 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 01 05:25:31.541490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:31.541303 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 01 05:25:32.559279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:32.548667 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 01 05:25:32.559279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:32.548734 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 01 05:25:33.545911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:33.545390 4425 patch_prober.go:29] interesting pod/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 01 05:25:33.545911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:33.545473 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 01 05:25:34.049402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.049366 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7"] Nov 01 05:25:34.100378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.100333 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9385bf-mdvf7"] Nov 01 05:25:34.547247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.547214 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:25:34.547651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.547629 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:25:34.548491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.548471 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:25:34.974120 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:34.974090 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd39080f-fc35-42a6-a0f8-ee7baf6c0c13" path="/var/lib/kubelet/pods/cd39080f-fc35-42a6-a0f8-ee7baf6c0c13/volumes" Nov 01 05:25:40.051779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.051747 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk"] Nov 01 05:25:40.079694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.079669 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-186a5ede-ltlpk"] Nov 01 05:25:40.949447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.949411 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:25:40.950054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950039 4425 cpu_manager.go:414] "containerMap: removing container" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" Nov 01 05:25:40.950105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950056 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" Nov 01 05:25:40.950105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950093 4425 cpu_manager.go:414] "containerMap: removing container" podUID="20d219a5-ab5f-4838-8526-6db70115ee2e" containerName="revoke-keys" Nov 01 05:25:40.950105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950098 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d219a5-ab5f-4838-8526-6db70115ee2e" containerName="revoke-keys" Nov 01 05:25:40.950200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950185 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="593c1831-d654-412a-b1bf-e125ed51154c" containerName="istio-proxy" Nov 01 05:25:40.950200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.950192 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d219a5-ab5f-4838-8526-6db70115ee2e" containerName="revoke-keys" Nov 01 05:25:40.955286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.955240 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:40.975661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.975632 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5af7c26-7ced-416b-9d69-64291a242fb8" path="/var/lib/kubelet/pods/b5af7c26-7ced-416b-9d69-64291a242fb8/volumes" Nov 01 05:25:40.991937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.991904 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:40.992064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.991945 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:40.992064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.992044 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:40.992167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:40.992125 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nr5\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.015181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.015149 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:25:41.061648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061582 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"routing-overlay\"" Nov 01 05:25:41.061648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061604 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a12c35\"" Nov 01 05:25:41.061648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061583 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"agc-cred-2a3644bb5550\"" Nov 01 05:25:41.061648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061633 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"payload-processing-external-model-e2e--57fe8cc0-dockercfg-8nd7t\"" Nov 01 05:25:41.062218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061751 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"openshift-service-ca.crt\"" Nov 01 05:25:41.062218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.061805 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"kube-root-ca.crt\"" Nov 01 05:25:41.093206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.093150 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.093416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.093212 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.093416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.093316 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.093416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.093404 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nr5\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.094304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.094280 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.094381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.094289 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.095869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.095851 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.156019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.155984 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nr5\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.270545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.270463 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.652625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.652592 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:25:41.652938 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:41.652912 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e5bf49_a979_4378_a316_287ef58f7ad1.slice/crio-2f525f491aeffc7d312750475efd17b3b17a54975df13583bd7bcee5daa2464f WatchSource:0}: Error finding container 2f525f491aeffc7d312750475efd17b3b17a54975df13583bd7bcee5daa2464f: Status 404 returned error can't find the container with id 2f525f491aeffc7d312750475efd17b3b17a54975df13583bd7bcee5daa2464f Nov 01 05:25:41.875606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.875571 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" event={"ID":"74e5bf49-a979-4378-a316-287ef58f7ad1","Type":"ContainerStarted","Data":"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440"} Nov 01 05:25:41.875774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.875633 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:41.875774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.875650 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" event={"ID":"74e5bf49-a979-4378-a316-287ef58f7ad1","Type":"ContainerStarted","Data":"2f525f491aeffc7d312750475efd17b3b17a54975df13583bd7bcee5daa2464f"} Nov 01 05:25:41.999240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:41.999125 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" podStartSLOduration=1.99911153 podStartE2EDuration="1.99911153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:41.994133883 +0000 UTC m=+1357.134722804" watchObservedRunningTime="2026-11-01 05:25:41.99911153 +0000 UTC m=+1357.139700471" Nov 01 05:25:42.266784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.266706 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:42.271051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.271034 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.309266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.309217 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.309447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.309303 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.309535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.309470 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.309596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.309553 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwpg\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.325474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325448 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\"" Nov 01 05:25:42.325709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325690 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-f8c698a71031\"" Nov 01 05:25:42.325796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325709 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\"" Nov 01 05:25:42.325796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325717 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\"" Nov 01 05:25:42.325796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325779 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-9394697c5d5d\"" Nov 01 05:25:42.325970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.325808 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\"" Nov 01 05:25:42.329029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.329007 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:42.375298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.375236 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-17306a2b8c16\"" Nov 01 05:25:42.375298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.375236 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\"" Nov 01 05:25:42.375298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.375266 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\"" Nov 01 05:25:42.411172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.411139 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.411381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.411202 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwpg\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.411381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.411334 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.411381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.411376 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.411969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.411938 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.412141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.412116 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.414807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.414781 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.474208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.474164 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwpg\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.585597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.585561 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:42.973199 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:42.973158 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20759fd7_80e3_4b10_8318_d7fe437ada98.slice/crio-10c028242cdce3cd338cf2e74f361779e37a40b4913a4b8f81b0f418856f2514 WatchSource:0}: Error finding container 10c028242cdce3cd338cf2e74f361779e37a40b4913a4b8f81b0f418856f2514: Status 404 returned error can't find the container with id 10c028242cdce3cd338cf2e74f361779e37a40b4913a4b8f81b0f418856f2514 Nov 01 05:25:42.977051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:42.977026 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:43.897272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:43.897224 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" event={"ID":"20759fd7-80e3-4b10-8318-d7fe437ada98","Type":"ContainerStarted","Data":"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8"} Nov 01 05:25:43.897762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:43.897536 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:43.897762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:43.897578 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" event={"ID":"20759fd7-80e3-4b10-8318-d7fe437ada98","Type":"ContainerStarted","Data":"10c028242cdce3cd338cf2e74f361779e37a40b4913a4b8f81b0f418856f2514"} Nov 01 05:25:44.018524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:44.018465 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" podStartSLOduration=2.01844463 podStartE2EDuration="2.01844463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:44.01828924 +0000 UTC m=+1359.158878159" watchObservedRunningTime="2026-11-01 05:25:44.01844463 +0000 UTC m=+1359.159033552" Nov 01 05:25:47.529772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:47.529735 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:52.530664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.530623 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" podUID="20759fd7-80e3-4b10-8318-d7fe437ada98" containerName="payload-processing" containerID="cri-o://4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8" gracePeriod=25 Nov 01 05:25:52.816310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.816282 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:52.889608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.889581 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:25:52.903301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903276 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\") pod \"20759fd7-80e3-4b10-8318-d7fe437ada98\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " Nov 01 05:25:52.903406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903394 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\") pod \"20759fd7-80e3-4b10-8318-d7fe437ada98\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " Nov 01 05:25:52.903516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903504 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\") pod \"20759fd7-80e3-4b10-8318-d7fe437ada98\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " Nov 01 05:25:52.903568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903557 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\") pod \"20759fd7-80e3-4b10-8318-d7fe437ada98\" (UID: \"20759fd7-80e3-4b10-8318-d7fe437ada98\") " Nov 01 05:25:52.903898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903873 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume" pod "20759fd7-80e3-4b10-8318-d7fe437ada98" (UID: "20759fd7-80e3-4b10-8318-d7fe437ada98"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:25:52.903988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.903947 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume" pod "20759fd7-80e3-4b10-8318-d7fe437ada98" (UID: "20759fd7-80e3-4b10-8318-d7fe437ada98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:25:52.904067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.904051 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:52.905424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.905405 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials" pod "20759fd7-80e3-4b10-8318-d7fe437ada98" (UID: "20759fd7-80e3-4b10-8318-d7fe437ada98"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:25:52.905672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:52.905653 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg" pod "20759fd7-80e3-4b10-8318-d7fe437ada98" (UID: "20759fd7-80e3-4b10-8318-d7fe437ada98"). InnerVolumeSpecName "kube-api-access-2lwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:25:53.005343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.005309 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20759fd7-80e3-4b10-8318-d7fe437ada98-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:53.005343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.005343 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:53.005785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.005359 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwpg\" (UniqueName: \"kubernetes.io/projected/20759fd7-80e3-4b10-8318-d7fe437ada98-kube-api-access-2lwpg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:25:53.005785 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.005451 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-2a3644bb5550: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:53.005785 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.005466 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:53.005785 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.005543 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials podName:74e5bf49-a979-4378-a316-287ef58f7ad1 nodeName:}" failed. No retries permitted until 2026-11-01 05:25:53.505522953 +0000 UTC m=+1368.646111869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1") : secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:53.017951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.017919 4425 generic.go:356] "Generic (PLEG): container finished" podID="20759fd7-80e3-4b10-8318-d7fe437ada98" containerID="4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8" exitCode=0 Nov 01 05:25:53.018086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.017966 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" Nov 01 05:25:53.018086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.017970 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" event={"ID":"20759fd7-80e3-4b10-8318-d7fe437ada98","Type":"ContainerDied","Data":"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8"} Nov 01 05:25:53.018086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.018012 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc" event={"ID":"20759fd7-80e3-4b10-8318-d7fe437ada98","Type":"ContainerDied","Data":"10c028242cdce3cd338cf2e74f361779e37a40b4913a4b8f81b0f418856f2514"} Nov 01 05:25:53.018086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.018034 4425 scope.go:122] "RemoveContainer" containerID="4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8" Nov 01 05:25:53.030012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.029989 4425 scope.go:122] "RemoveContainer" containerID="4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8" Nov 01 05:25:53.030300 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.030265 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8\": container with ID starting with 4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8 not found: ID does not exist" containerID="4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8" Nov 01 05:25:53.030367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.030312 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8"} err="failed to get container status \"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8\": rpc error: code = NotFound desc = could not find container \"4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8\": container with ID starting with 4c4e4999481f1154fe13acdb15fc5065e67b21b23d08ce8dade4172881eac7b8 not found: ID does not exist" Nov 01 05:25:53.189456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.189422 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:53.196378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:53.196353 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-76ffq4szc"] Nov 01 05:25:53.512268 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.512173 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-2a3644bb5550: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:53.512268 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.512197 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:53.512268 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:53.512266 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials podName:74e5bf49-a979-4378-a316-287ef58f7ad1 nodeName:}" failed. No retries permitted until 2026-11-01 05:25:54.512235368 +0000 UTC m=+1369.652824278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1") : secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:54.525342 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:54.525302 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-2a3644bb5550: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:54.525342 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:54.525333 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:54.525792 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:54.525391 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials podName:74e5bf49-a979-4378-a316-287ef58f7ad1 nodeName:}" failed. No retries permitted until 2026-11-01 05:25:56.525375275 +0000 UTC m=+1371.665964188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1") : secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:54.974017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:54.973985 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20759fd7-80e3-4b10-8318-d7fe437ada98" path="/var/lib/kubelet/pods/20759fd7-80e3-4b10-8318-d7fe437ada98/volumes" Nov 01 05:25:56.074632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.074598 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:25:56.075240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.075226 4425 cpu_manager.go:414] "containerMap: removing container" podUID="20759fd7-80e3-4b10-8318-d7fe437ada98" containerName="payload-processing" Nov 01 05:25:56.075304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.075242 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20759fd7-80e3-4b10-8318-d7fe437ada98" containerName="payload-processing" Nov 01 05:25:56.075392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.075382 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="20759fd7-80e3-4b10-8318-d7fe437ada98" containerName="payload-processing" Nov 01 05:25:56.079105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.079089 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.166138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.166104 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:25:56.184280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.183683 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ca7a55-openshift-default-dockercfg-fcc2q\"" Nov 01 05:25:56.193297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.190362 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:25:56.252653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.252610 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.252890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.252712 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.252890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.252860 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.252902 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.252924 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.253044 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.253091 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.253267 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77z2p\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.253353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.253329 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.354838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354768 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77z2p\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.354838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354854 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354897 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354952 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354976 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.354997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355053 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355091 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355323 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355400 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355534 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355557 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.355793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.355771 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.357512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.357472 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.357698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.357679 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.418682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.418650 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.418944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.418919 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77z2p\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\") pod \"e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:56.559438 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:56.559404 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-2a3644bb5550: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:56.559438 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:56.559430 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc: secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:56.559651 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:56.559496 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials podName:74e5bf49-a979-4378-a316-287ef58f7ad1 nodeName:}" failed. No retries permitted until 2026-11-01 05:26:00.559481556 +0000 UTC m=+1375.700070453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1") : secret "agc-cred-2a3644bb5550" not found Nov 01 05:25:56.692332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:56.692233 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:57.003116 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:25:57.002997 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:25:57.053511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.053464 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh"] Nov 01 05:25:57.057507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.057480 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4adcec-tv8qh"] Nov 01 05:25:57.213463 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:25:57.213434 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb211b3_12af_4e82_9bc5_6a0478a55fbe.slice/crio-482928ee4f54668c70eb3e54f0566c42be3f5e4f32764166808a0ff1b8750df0 WatchSource:0}: Error finding container 482928ee4f54668c70eb3e54f0566c42be3f5e4f32764166808a0ff1b8750df0: Status 404 returned error can't find the container with id 482928ee4f54668c70eb3e54f0566c42be3f5e4f32764166808a0ff1b8750df0 Nov 01 05:25:57.214633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.214612 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:25:57.216206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.216173 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:57.216330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.216298 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:57.216395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:57.216351 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:25:58.064094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.064059 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerStarted","Data":"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0"} Nov 01 05:25:58.064342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.064112 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerStarted","Data":"482928ee4f54668c70eb3e54f0566c42be3f5e4f32764166808a0ff1b8750df0"} Nov 01 05:25:58.243828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.243767 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podStartSLOduration=2.243748283 podStartE2EDuration="2.243748283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:58.237970696 +0000 UTC m=+1373.378559639" watchObservedRunningTime="2026-11-01 05:25:58.243748283 +0000 UTC m=+1373.384337201" Nov 01 05:25:58.692555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.692529 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:58.697560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.697537 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:58.974595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:58.974514 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2496da3e-da85-4f81-abd1-662ac351d46b" path="/var/lib/kubelet/pods/2496da3e-da85-4f81-abd1-662ac351d46b/volumes" Nov 01 05:25:59.078293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:59.078214 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:25:59.079640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:25:59.079618 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:26:00.615597 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:00.615568 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-2a3644bb5550: secret "agc-cred-2a3644bb5550" not found Nov 01 05:26:00.615597 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:00.615596 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc: secret "agc-cred-2a3644bb5550" not found Nov 01 05:26:00.616042 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:00.615689 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials podName:74e5bf49-a979-4378-a316-287ef58f7ad1 nodeName:}" failed. No retries permitted until 2026-11-01 05:26:08.615672151 +0000 UTC m=+1383.756261071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1") : secret "agc-cred-2a3644bb5550" not found Nov 01 05:26:01.167897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.167851 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" podUID="74e5bf49-a979-4378-a316-287ef58f7ad1" containerName="payload-processing" containerID="cri-o://7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440" gracePeriod=25 Nov 01 05:26:01.439606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.439576 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:26:01.530674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.530636 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\") pod \"74e5bf49-a979-4378-a316-287ef58f7ad1\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " Nov 01 05:26:01.530875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.530781 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\") pod \"74e5bf49-a979-4378-a316-287ef58f7ad1\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " Nov 01 05:26:01.530875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.530830 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\") pod \"74e5bf49-a979-4378-a316-287ef58f7ad1\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " Nov 01 05:26:01.530979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.530875 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\") pod \"74e5bf49-a979-4378-a316-287ef58f7ad1\" (UID: \"74e5bf49-a979-4378-a316-287ef58f7ad1\") " Nov 01 05:26:01.531707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.531624 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume" pod "74e5bf49-a979-4378-a316-287ef58f7ad1" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:26:01.531973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.531955 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume" pod "74e5bf49-a979-4378-a316-287ef58f7ad1" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:26:01.544391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.540434 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials" pod "74e5bf49-a979-4378-a316-287ef58f7ad1" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:26:01.544391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.540527 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5" pod "74e5bf49-a979-4378-a316-287ef58f7ad1" (UID: "74e5bf49-a979-4378-a316-287ef58f7ad1"). InnerVolumeSpecName "kube-api-access-t8nr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:26:01.633370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.633332 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:01.633370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.633373 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8nr5\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-kube-api-access-t8nr5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:01.633802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.633389 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74e5bf49-a979-4378-a316-287ef58f7ad1-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:01.633802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:01.633404 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74e5bf49-a979-4378-a316-287ef58f7ad1-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:02.101268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.101222 4425 generic.go:356] "Generic (PLEG): container finished" podID="74e5bf49-a979-4378-a316-287ef58f7ad1" containerID="7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440" exitCode=0 Nov 01 05:26:02.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.101298 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" Nov 01 05:26:02.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.101315 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" event={"ID":"74e5bf49-a979-4378-a316-287ef58f7ad1","Type":"ContainerDied","Data":"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440"} Nov 01 05:26:02.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.101385 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc" event={"ID":"74e5bf49-a979-4378-a316-287ef58f7ad1","Type":"ContainerDied","Data":"2f525f491aeffc7d312750475efd17b3b17a54975df13583bd7bcee5daa2464f"} Nov 01 05:26:02.101479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.101415 4425 scope.go:122] "RemoveContainer" containerID="7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440" Nov 01 05:26:02.113199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.113163 4425 scope.go:122] "RemoveContainer" containerID="7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440" Nov 01 05:26:02.113625 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:02.113604 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440\": container with ID starting with 7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440 not found: ID does not exist" containerID="7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440" Nov 01 05:26:02.113670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.113634 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440"} err="failed to get container status \"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440\": rpc error: code = NotFound desc = could not find container \"7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440\": container with ID starting with 7f20744e90f3cebd64286d56a8a0bd949aba78f87a556dc12150201917536440 not found: ID does not exist" Nov 01 05:26:02.271545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.271510 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:26:02.317432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.317396 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-76f4rm5xc"] Nov 01 05:26:02.974403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:02.974371 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e5bf49-a979-4378-a316-287ef58f7ad1" path="/var/lib/kubelet/pods/74e5bf49-a979-4378-a316-287ef58f7ad1/volumes" Nov 01 05:26:05.601683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.601650 4425 scope.go:122] "RemoveContainer" containerID="3041f0e6e24325747ad3343661d7111a6a1600c2bd5ff171eb4f4a779e47f05b" Nov 01 05:26:05.616422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.616403 4425 scope.go:122] "RemoveContainer" containerID="be180caef74c8749a8787dd0f98dfffba9af281fa15715b6c3d13ecbdf219ce3" Nov 01 05:26:05.629906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.629865 4425 scope.go:122] "RemoveContainer" containerID="1f99a623f654901ce2768c49516b9048e0de28cf6e50195189516e598c1e0c87" Nov 01 05:26:05.674396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.674368 4425 scope.go:122] "RemoveContainer" containerID="ba21f1992f6385a2975234f233e9f11c11eb287afc1c8b47faef2a5db8475bf4" Nov 01 05:26:05.709009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.708987 4425 scope.go:122] "RemoveContainer" containerID="dd66a0aea9cba83c1cda9867dd8d61cb3ac51995bae3ef7ce0bd06aa6b529e23" Nov 01 05:26:05.721237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.721211 4425 scope.go:122] "RemoveContainer" containerID="6cabbc5f7c99edeeb5c6f6ca96a163b43f490609728e843e7a28fe444aa2f2d6" Nov 01 05:26:05.731770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.731747 4425 scope.go:122] "RemoveContainer" containerID="3601269034a1b78c1760d73bc6d8f7f4981044ed184ef029678ec3c0415c6a99" Nov 01 05:26:05.742242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:05.742224 4425 scope.go:122] "RemoveContainer" containerID="58c44d9cc71d8e46176658f746ddd512c8f9065bdb855d629f1238e56a257c28" Nov 01 05:26:06.893718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.893685 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:26:06.894366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.894351 4425 cpu_manager.go:414] "containerMap: removing container" podUID="74e5bf49-a979-4378-a316-287ef58f7ad1" containerName="payload-processing" Nov 01 05:26:06.894366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.894366 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e5bf49-a979-4378-a316-287ef58f7ad1" containerName="payload-processing" Nov 01 05:26:06.894478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.894468 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e5bf49-a979-4378-a316-287ef58f7ad1" containerName="payload-processing" Nov 01 05:26:06.899889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.899872 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:06.991998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.991966 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 05:26:06.992205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.991966 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 05:26:06.992205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.992023 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 05:26:06.992205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:06.992123 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44fmh\"" Nov 01 05:26:07.002227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.002194 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:26:07.005588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005560 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.005759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005649 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.005759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005708 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.005850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005758 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.005901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005843 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.005994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.005974 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpr7\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.006118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.006100 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.107213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.107178 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpr7\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.107458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.107301 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.107458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.107347 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.108009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.107953 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.108156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.108063 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.108156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.108115 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.108310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.108278 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.109060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.108835 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.116736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.116707 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.116959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.116933 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.116959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.116951 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.120591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.120568 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.171149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.171074 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpr7\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.171149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.171101 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gwt5l\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.213364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.213333 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:07.647569 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:26:07.647532 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd64301_b5b2_4179_82f7_63f7fc574ab2.slice/crio-30981fd7d54a11bc78bfd82a213eb83779a909d92abaf510e8cf01003fdf2270 WatchSource:0}: Error finding container 30981fd7d54a11bc78bfd82a213eb83779a909d92abaf510e8cf01003fdf2270: Status 404 returned error can't find the container with id 30981fd7d54a11bc78bfd82a213eb83779a909d92abaf510e8cf01003fdf2270 Nov 01 05:26:07.648182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.648156 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:26:07.649816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:07.649797 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:26:08.170307 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:08.154091 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" event={"ID":"afd64301-b5b2-4179-82f7-63f7fc574ab2","Type":"ContainerStarted","Data":"30981fd7d54a11bc78bfd82a213eb83779a909d92abaf510e8cf01003fdf2270"} Nov 01 05:26:09.489021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.488981 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r"] Nov 01 05:26:09.495506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.495484 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.541866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.541832 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdpk\" (UniqueName: \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.542063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.541963 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.542063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.541995 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.546910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.546879 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r"] Nov 01 05:26:09.592776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.592747 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:26:09.592958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.592838 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:26:09.635598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.635569 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:26:09.635824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.635646 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:26:09.644156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.644116 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdpk\" (UniqueName: \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.644366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.644231 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.644366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.644296 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.646294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.646238 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.647704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.647680 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.709312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.709286 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdpk\" (UniqueName: \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\") pod \"maas-api-revoke-sub-f5c87e06-d4s7r\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:09.816877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:09.816852 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:10.199293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:10.199247 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r"] Nov 01 05:26:10.201433 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:26:10.201404 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfe56ff_f7c8_43cb_80f8_808f3f460da4.slice/crio-ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b WatchSource:0}: Error finding container ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b: Status 404 returned error can't find the container with id ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b Nov 01 05:26:11.193355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:11.193295 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" event={"ID":"ccfe56ff-f7c8-43cb-80f8-808f3f460da4","Type":"ContainerStarted","Data":"5d2bf44d3c04ca5e50dcf2aa263f0ca821d3749c858599935db5bac5bb44fe34"} Nov 01 05:26:11.193859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:11.193418 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" event={"ID":"ccfe56ff-f7c8-43cb-80f8-808f3f460da4","Type":"ContainerStarted","Data":"ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b"} Nov 01 05:26:11.312430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:11.312366 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" podStartSLOduration=2.312346133 podStartE2EDuration="2.312346133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:11.311626678 +0000 UTC m=+1386.452215625" watchObservedRunningTime="2026-11-01 05:26:11.312346133 +0000 UTC m=+1386.452935053" Nov 01 05:26:11.963496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:11.963463 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:26:11.963564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:11.963541 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:26:12.202537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:12.202503 4425 generic.go:356] "Generic (PLEG): container finished" podID="ccfe56ff-f7c8-43cb-80f8-808f3f460da4" containerID="5d2bf44d3c04ca5e50dcf2aa263f0ca821d3749c858599935db5bac5bb44fe34" exitCode=0 Nov 01 05:26:12.202930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:12.202586 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" event={"ID":"ccfe56ff-f7c8-43cb-80f8-808f3f460da4","Type":"ContainerDied","Data":"5d2bf44d3c04ca5e50dcf2aa263f0ca821d3749c858599935db5bac5bb44fe34"} Nov 01 05:26:12.204420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:12.204392 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" event={"ID":"afd64301-b5b2-4179-82f7-63f7fc574ab2","Type":"ContainerStarted","Data":"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e"} Nov 01 05:26:12.204535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:12.204457 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:12.451436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:12.451359 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" podStartSLOduration=2.13914277 podStartE2EDuration="6.45134242s" totalImagesPullingTime="4.31219965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:12.447517846 +0000 UTC m=+1387.588106787" watchObservedRunningTime="2026-11-01 05:26:12.45134242 +0000 UTC m=+1387.591931337" Nov 01 05:26:13.218895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.218865 4425 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gwt5l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:26:13.219395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.218935 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:26:13.435172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.435144 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:13.505998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.505871 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\" (UniqueName: \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\") pod \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " Nov 01 05:26:13.505998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.505981 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\") pod \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " Nov 01 05:26:13.506353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.506144 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\") pod \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\" (UID: \"ccfe56ff-f7c8-43cb-80f8-808f3f460da4\") " Nov 01 05:26:13.507424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.507017 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca" pod "ccfe56ff-f7c8-43cb-80f8-808f3f460da4" (UID: "ccfe56ff-f7c8-43cb-80f8-808f3f460da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:26:13.548280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.544432 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca" pod "ccfe56ff-f7c8-43cb-80f8-808f3f460da4" (UID: "ccfe56ff-f7c8-43cb-80f8-808f3f460da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:26:13.548280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.544580 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk" pod "ccfe56ff-f7c8-43cb-80f8-808f3f460da4" (UID: "ccfe56ff-f7c8-43cb-80f8-808f3f460da4"). InnerVolumeSpecName "kube-api-access-sbdpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:26:13.616273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.610619 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbdpk\" (UniqueName: \"kubernetes.io/projected/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-kube-api-access-sbdpk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:13.616273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.610653 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:13.616273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:13.610669 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccfe56ff-f7c8-43cb-80f8-808f3f460da4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:26:14.082495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.078444 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp"] Nov 01 05:26:14.084773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.084730 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c04b93-9cqfp"] Nov 01 05:26:14.231439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.231400 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" event={"ID":"ccfe56ff-f7c8-43cb-80f8-808f3f460da4","Type":"ContainerDied","Data":"ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b"} Nov 01 05:26:14.231812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.231468 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3b0e78563f432971360292203894f362b3072bab0af914ee4ff76560eeb92b" Nov 01 05:26:14.231812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.231544 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r" Nov 01 05:26:14.975331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:14.975298 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6638d2-6339-47ab-8f5e-361e51ae826b" path="/var/lib/kubelet/pods/3f6638d2-6339-47ab-8f5e-361e51ae826b/volumes" Nov 01 05:26:16.217621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:16.217586 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:26:23.321975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.321946 4425 generic.go:356] "Generic (PLEG): container finished" podID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerID="84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a" exitCode=0 Nov 01 05:26:23.322420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.322021 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerDied","Data":"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a"} Nov 01 05:26:23.322490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.322476 4425 scope.go:122] "RemoveContainer" containerID="84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a" Nov 01 05:26:23.323924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.323898 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:26:23.323984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.323965 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:26:23.324017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:23.323999 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:26:24.331281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:24.331223 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerStarted","Data":"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa"} Nov 01 05:26:24.657800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:24.657698 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:26:24.658163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:24.658132 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 01 05:26:24.658429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:24.658403 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 01 05:26:25.666765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:25.666723 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 01 05:26:25.667317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:25.666790 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 01 05:26:26.652288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:26.649681 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 01 05:26:26.652288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:26.649761 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 01 05:26:26.973364 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:26.973242 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:26:27.656279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:27.648668 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 01 05:26:27.656279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:27.648753 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 01 05:26:28.649694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:28.649657 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 01 05:26:28.655111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:28.655076 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 01 05:26:29.652030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:29.652000 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:26:29.652517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:29.652504 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:26:29.653149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:29.653131 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 05:26:30.049443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:30.049369 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z"] Nov 01 05:26:30.053275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:30.053238 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65604218-rb48z"] Nov 01 05:26:30.974960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:30.974926 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae4fea1-3707-4fe3-bc87-753dd4f219b4" path="/var/lib/kubelet/pods/aae4fea1-3707-4fe3-bc87-753dd4f219b4/volumes" Nov 01 05:26:37.048329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:37.048297 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr"] Nov 01 05:26:37.098938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:37.098899 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e695195-2qqpr"] Nov 01 05:26:38.974345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:38.974311 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6eb8dd2-8b82-46b4-8a54-065b1a5a251a" path="/var/lib/kubelet/pods/b6eb8dd2-8b82-46b4-8a54-065b1a5a251a/volumes" Nov 01 05:26:56.954566 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:26:56.954515 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:26:58.549660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.549622 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562"] Nov 01 05:26:58.550353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.550337 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ccfe56ff-f7c8-43cb-80f8-808f3f460da4" containerName="revoke-keys" Nov 01 05:26:58.550411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.550354 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfe56ff-f7c8-43cb-80f8-808f3f460da4" containerName="revoke-keys" Nov 01 05:26:58.550467 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.550458 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfe56ff-f7c8-43cb-80f8-808f3f460da4" containerName="revoke-keys" Nov 01 05:26:58.553801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.553784 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.611651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.611618 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562"] Nov 01 05:26:58.656520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.656489 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:26:58.656732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.656713 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:26:58.656868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.656851 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:26:58.659383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.659366 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:26:58.692905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.692869 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lpn\" (UniqueName: \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.693104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.693018 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.693286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.693236 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.795146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.795103 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lpn\" (UniqueName: \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.795360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.795211 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.795444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.795395 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.796141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.796107 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.798009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.797983 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.860602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.860535 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lpn\" (UniqueName: \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:58.868780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:58.868557 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:26:59.327548 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:26:59.327521 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d104cc_7a59_42ef_b35a_b07208246e63.slice/crio-f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e WatchSource:0}: Error finding container f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e: Status 404 returned error can't find the container with id f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e Nov 01 05:26:59.328421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:59.328399 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562"] Nov 01 05:26:59.679265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:26:59.679209 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" event={"ID":"74d104cc-7a59-42ef-b35a-b07208246e63","Type":"ContainerStarted","Data":"f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e"} Nov 01 05:27:00.700156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:00.700051 4425 generic.go:356] "Generic (PLEG): container finished" podID="74d104cc-7a59-42ef-b35a-b07208246e63" containerID="7bad98cbf61b3d733c4f231494297968d506467503856a9a28940fbb9d468f42" exitCode=0 Nov 01 05:27:00.700156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:00.700136 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" event={"ID":"74d104cc-7a59-42ef-b35a-b07208246e63","Type":"ContainerDied","Data":"7bad98cbf61b3d733c4f231494297968d506467503856a9a28940fbb9d468f42"} Nov 01 05:27:01.919220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:01.919187 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:27:02.052663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.041825 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\") pod \"74d104cc-7a59-42ef-b35a-b07208246e63\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " Nov 01 05:27:02.052663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.041892 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\") pod \"74d104cc-7a59-42ef-b35a-b07208246e63\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " Nov 01 05:27:02.052663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.042080 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\" (UniqueName: \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\") pod \"74d104cc-7a59-42ef-b35a-b07208246e63\" (UID: \"74d104cc-7a59-42ef-b35a-b07208246e63\") " Nov 01 05:27:02.053718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.053684 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca" pod "74d104cc-7a59-42ef-b35a-b07208246e63" (UID: "74d104cc-7a59-42ef-b35a-b07208246e63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:27:02.099084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.099044 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn" pod "74d104cc-7a59-42ef-b35a-b07208246e63" (UID: "74d104cc-7a59-42ef-b35a-b07208246e63"). InnerVolumeSpecName "kube-api-access-p8lpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:27:02.099555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.099512 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca" pod "74d104cc-7a59-42ef-b35a-b07208246e63" (UID: "74d104cc-7a59-42ef-b35a-b07208246e63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:27:02.148387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.148331 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:02.149107 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.149088 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d104cc-7a59-42ef-b35a-b07208246e63-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:02.149291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.149272 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lpn\" (UniqueName: \"kubernetes.io/projected/74d104cc-7a59-42ef-b35a-b07208246e63-kube-api-access-p8lpn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:02.718696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.718667 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" Nov 01 05:27:02.718905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.718696 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562" event={"ID":"74d104cc-7a59-42ef-b35a-b07208246e63","Type":"ContainerDied","Data":"f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e"} Nov 01 05:27:02.718905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:02.718758 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65e3fa01f6eb7cb4977f49718ab80e213e1cc36e6d4c6bb6a5345d7abf9f55e" Nov 01 05:27:05.050871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.050776 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz"] Nov 01 05:27:05.102547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.102508 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e67ede9-bvxnz"] Nov 01 05:27:05.822913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.822873 4425 scope.go:122] "RemoveContainer" containerID="6fa1a6f0d26f756f4746873727091b7145dc758820f5b0a538b30b455f8c645b" Nov 01 05:27:05.836811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.836784 4425 scope.go:122] "RemoveContainer" containerID="45ab2f926f90c52f6645e5b18438173fe3383a9af1c1118eabff8babaf41b261" Nov 01 05:27:05.848400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.848372 4425 scope.go:122] "RemoveContainer" containerID="a55b2aea3c2d0466304c60ce6c86291787a732a083a36b4d30bcc11288593132" Nov 01 05:27:05.863758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.863453 4425 scope.go:122] "RemoveContainer" containerID="9b5f5ff0c70fd983b4880bc43bfc43ce24277598efa1776203451a27179ebe0d" Nov 01 05:27:05.878410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.878383 4425 scope.go:122] "RemoveContainer" containerID="c665cb831c12b7eb78612320581cd435fe6b18e3b92852f9fa5a4ab459869933" Nov 01 05:27:05.889150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:05.889108 4425 scope.go:122] "RemoveContainer" containerID="cc95224ec5b7fd15e4f9268d7ce0c98da6b1388361c0e64ea1ae0ed5e51f609a" Nov 01 05:27:06.975508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:06.975468 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17643fb3-e251-4a88-b7aa-06a591d4814a" path="/var/lib/kubelet/pods/17643fb3-e251-4a88-b7aa-06a591d4814a/volumes" Nov 01 05:27:08.785369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.785333 4425 generic.go:356] "Generic (PLEG): container finished" podID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerID="b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0" exitCode=0 Nov 01 05:27:08.785902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.785410 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerDied","Data":"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0"} Nov 01 05:27:08.785994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.785921 4425 scope.go:122] "RemoveContainer" containerID="b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0" Nov 01 05:27:08.787458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.787431 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:27:08.787543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.787496 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:27:08.787543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:08.787528 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:27:09.795512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:09.795473 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerStarted","Data":"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b"} Nov 01 05:27:10.692454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:10.692417 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:27:10.693615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:10.693585 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 05:27:10.693739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:10.693660 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 05:27:11.071967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:11.071929 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr"] Nov 01 05:27:11.118353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:11.118313 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e2b2e85-2vqkr"] Nov 01 05:27:11.693207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:11.693170 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 05:27:11.693462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:11.693228 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 05:27:12.692817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:12.692778 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 05:27:12.693264 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:12.692845 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 05:27:12.975400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:12.975321 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ecb737-5d04-414b-af48-13250ec0f214" path="/var/lib/kubelet/pods/68ecb737-5d04-414b-af48-13250ec0f214/volumes" Nov 01 05:27:13.692877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:13.692842 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 05:27:13.693420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:13.692909 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 05:27:14.078092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:14.078062 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:27:14.693286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:14.693239 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 05:27:14.693776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:14.693318 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 05:27:15.698608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:15.698579 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:27:15.699897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:15.699874 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:27:17.810948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:17.810915 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562"] Nov 01 05:27:17.864666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:17.864623 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2864b6-dq562"] Nov 01 05:27:18.047218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:18.047183 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:27:18.047525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:18.047495 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" containerID="cri-o://7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82" gracePeriod=30 Nov 01 05:27:18.974590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:18.974560 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d104cc-7a59-42ef-b35a-b07208246e63" path="/var/lib/kubelet/pods/74d104cc-7a59-42ef-b35a-b07208246e63/volumes" Nov 01 05:27:23.335869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.335835 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:27:23.392022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.391978 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392041 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392089 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392149 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392235 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392317 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392326 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:27:23.392435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392350 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392433 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392462 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\") pod \"aa71ec23-30b9-460c-b9b6-e054d0640203\" (UID: \"aa71ec23-30b9-460c-b9b6-e054d0640203\") " Nov 01 05:27:23.392680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392510 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:27:23.392680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392599 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:27:23.392833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.392674 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:27:23.393140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.393100 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:27:23.393275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.393219 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.393275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.393243 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.393371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.393277 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.393371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.393292 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.395187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.395157 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:27:23.395340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.395268 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:27:23.395548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.395533 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:27:23.395701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.395676 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx" pod "aa71ec23-30b9-460c-b9b6-e054d0640203" (UID: "aa71ec23-30b9-460c-b9b6-e054d0640203"). InnerVolumeSpecName "kube-api-access-rbqdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:27:23.494089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.494051 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbqdx\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-kube-api-access-rbqdx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.494089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.494081 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.494089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.494092 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa71ec23-30b9-460c-b9b6-e054d0640203-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.494372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.494101 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.494372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.494130 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa71ec23-30b9-460c-b9b6-e054d0640203-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:27:23.920871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.920836 4425 generic.go:356] "Generic (PLEG): container finished" podID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerID="7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82" exitCode=0 Nov 01 05:27:23.921073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.920892 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" Nov 01 05:27:23.921073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.920914 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerDied","Data":"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82"} Nov 01 05:27:23.921073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.920990 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6" event={"ID":"aa71ec23-30b9-460c-b9b6-e054d0640203","Type":"ContainerDied","Data":"3e625a11b5907f38c2439dd6ad75113fb99084ab363a5593a504b7eec05f16e4"} Nov 01 05:27:23.921073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.921012 4425 scope.go:122] "RemoveContainer" containerID="7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82" Nov 01 05:27:23.932482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.932458 4425 scope.go:122] "RemoveContainer" containerID="78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1" Nov 01 05:27:23.942505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.942484 4425 scope.go:122] "RemoveContainer" containerID="7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82" Nov 01 05:27:23.942837 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:27:23.942817 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82\": container with ID starting with 7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82 not found: ID does not exist" containerID="7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82" Nov 01 05:27:23.942888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.942846 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82"} err="failed to get container status \"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82\": rpc error: code = NotFound desc = could not find container \"7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82\": container with ID starting with 7c0e7f3ad19c256dd38767e23bb47b5b3186c7c582176f1294dc9991b813ac82 not found: ID does not exist" Nov 01 05:27:23.942888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.942864 4425 scope.go:122] "RemoveContainer" containerID="78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1" Nov 01 05:27:23.943139 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:27:23.943117 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1\": container with ID starting with 78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1 not found: ID does not exist" containerID="78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1" Nov 01 05:27:23.943185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:23.943146 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1"} err="failed to get container status \"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1\": rpc error: code = NotFound desc = could not find container \"78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1\": container with ID starting with 78140f4c782d6b9a5734c6d60a316a1f3b7e01fb26075d3571aa05a8cd71c4e1 not found: ID does not exist" Nov 01 05:27:24.049994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.049960 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4"] Nov 01 05:27:24.053829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.053802 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0345facf-zxpq4"] Nov 01 05:27:24.091931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.091901 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:27:24.101296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.101241 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2864b6-openshift-default-864f5c8fbc-8hck6"] Nov 01 05:27:24.973923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.973890 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253e0d15-3c47-4e0e-a213-f3c866384ab2" path="/var/lib/kubelet/pods/253e0d15-3c47-4e0e-a213-f3c866384ab2/volumes" Nov 01 05:27:24.974422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:24.974405 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" path="/var/lib/kubelet/pods/aa71ec23-30b9-460c-b9b6-e054d0640203/volumes" Nov 01 05:27:26.958537 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:27:26.958503 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:27:27.047733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:27.047698 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc"] Nov 01 05:27:27.098582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:27.098546 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b1b6b51-dsrbc"] Nov 01 05:27:28.973738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:28.973705 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735de069-0a10-4c8e-8aea-8349e53ba1c0" path="/var/lib/kubelet/pods/735de069-0a10-4c8e-8aea-8349e53ba1c0/volumes" Nov 01 05:27:40.049210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:40.049176 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr"] Nov 01 05:27:40.100268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:40.100232 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b253a1c-bz8rr"] Nov 01 05:27:40.973843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:40.973804 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178b7008-31b1-4f59-a219-282693739a7c" path="/var/lib/kubelet/pods/178b7008-31b1-4f59-a219-282693739a7c/volumes" Nov 01 05:27:44.057732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:44.057701 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg"] Nov 01 05:27:44.113518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:44.113474 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc035ea1-zb7gg"] Nov 01 05:27:44.976109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:44.976073 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b4da31-4330-432d-afe9-b10ab99fa14d" path="/var/lib/kubelet/pods/b3b4da31-4330-432d-afe9-b10ab99fa14d/volumes" Nov 01 05:27:54.048202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:54.048111 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g"] Nov 01 05:27:54.099377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:54.099342 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-821ecd17-wbh8g"] Nov 01 05:27:54.978303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:27:54.978236 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ba969c-c314-4c1b-8125-e64a722efc5d" path="/var/lib/kubelet/pods/b4ba969c-c314-4c1b-8125-e64a722efc5d/volumes" Nov 01 05:27:56.948029 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:27:56.947991 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:28:00.047732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:00.047697 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9"] Nov 01 05:28:00.098564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:00.098519 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dff679c-dx8j9"] Nov 01 05:28:00.974847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:00.974803 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7ae036-6883-4b89-b433-17b9e6addd63" path="/var/lib/kubelet/pods/5f7ae036-6883-4b89-b433-17b9e6addd63/volumes" Nov 01 05:28:05.956419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:05.956390 4425 scope.go:122] "RemoveContainer" containerID="3778870fe14267d2e87c546b0e4211fb1a722915bdec12c7d85de5cdbb38dd58" Nov 01 05:28:05.967169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:05.967151 4425 scope.go:122] "RemoveContainer" containerID="03da23c2f7647c304214749e8c121fa22eb5c78d92f5acffe2d6c5c120305fa5" Nov 01 05:28:05.976486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:05.976462 4425 scope.go:122] "RemoveContainer" containerID="a5004c6178f0edc66774082b4d5aa90edc1bedba32c296052c2369a3d112d5d0" Nov 01 05:28:05.985991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:05.985969 4425 scope.go:122] "RemoveContainer" containerID="69cb6a1453a453ad02b33cad4c4f08064c58bbc3399cfd04162e176c7a75f6fa" Nov 01 05:28:13.051635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:13.051602 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk"] Nov 01 05:28:13.102657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:13.102626 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef0173bc-f2vmk"] Nov 01 05:28:14.974239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:14.974206 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70783ac9-9aac-4ba5-a941-00c25c4f68ce" path="/var/lib/kubelet/pods/70783ac9-9aac-4ba5-a941-00c25c4f68ce/volumes" Nov 01 05:28:22.048034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:22.047997 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz"] Nov 01 05:28:22.098672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:22.098640 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ed1d2d-hhlwz"] Nov 01 05:28:22.974449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:22.974419 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e36600-22d7-49c7-8cd0-903dce78e7d3" path="/var/lib/kubelet/pods/00e36600-22d7-49c7-8cd0-903dce78e7d3/volumes" Nov 01 05:28:26.947524 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:28:26.947488 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:28:33.049574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:33.049536 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v"] Nov 01 05:28:33.100389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:33.100353 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f54590-zmj4v"] Nov 01 05:28:34.974110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:34.974079 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3" path="/var/lib/kubelet/pods/bb4b338b-5f8e-48eb-9bab-2ce1ce8f63a3/volumes" Nov 01 05:28:41.048178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:41.048144 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct"] Nov 01 05:28:41.099073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:41.099041 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1657372f-wtlct"] Nov 01 05:28:42.973626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:42.973592 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9157b2-81a4-4f9f-ae1d-4be42ec6559f" path="/var/lib/kubelet/pods/7c9157b2-81a4-4f9f-ae1d-4be42ec6559f/volumes" Nov 01 05:28:49.051809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:49.051776 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294"] Nov 01 05:28:49.102579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:49.102547 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908f41d5-n8294"] Nov 01 05:28:50.973612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:50.973576 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae36317-fff1-44e2-9097-cb863e865901" path="/var/lib/kubelet/pods/9ae36317-fff1-44e2-9097-cb863e865901/volumes" Nov 01 05:28:52.051580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:52.051541 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w"] Nov 01 05:28:52.102426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:52.102393 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6819094b-rnv6w"] Nov 01 05:28:52.974077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:28:52.974040 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9" path="/var/lib/kubelet/pods/7d7166c2-4d0d-4c9d-b575-b0b9a4b75ed9/volumes" Nov 01 05:28:56.949912 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:28:56.949867 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:29:05.048503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:05.048469 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv"] Nov 01 05:29:05.099066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:05.099034 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a35340a-zmjfv"] Nov 01 05:29:06.037639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.037608 4425 scope.go:122] "RemoveContainer" containerID="50d5548829f3ac911f3087a8162d16857116040ac13ad974ffd38a27ef7dbcd9" Nov 01 05:29:06.047864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.047844 4425 scope.go:122] "RemoveContainer" containerID="3a40c45847da2483465a19273d3f4efac1b084012ec83fedfaaf37ccdfbfbbdc" Nov 01 05:29:06.057453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.057431 4425 scope.go:122] "RemoveContainer" containerID="ba296c5729933e1a9dfa1511772f0e84d6f389e6c4d15174e163d3c1c3d5da56" Nov 01 05:29:06.070499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.070477 4425 scope.go:122] "RemoveContainer" containerID="697928dc385f065633009db522d7c805685d6b0e68931fa16de7e4cd6e20c3de" Nov 01 05:29:06.079723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.079697 4425 scope.go:122] "RemoveContainer" containerID="5966607157daf123b13d9a65bee830115c34359e9fb9ff199b23db3a7576f857" Nov 01 05:29:06.088465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.088440 4425 scope.go:122] "RemoveContainer" containerID="1d02235a1763cc0dd3ef7663314b945473d66d56e838d2b253baa240d15de9b6" Nov 01 05:29:06.097162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.097140 4425 scope.go:122] "RemoveContainer" containerID="d246b463fb123d42e9d17ab5c2d7f738704f70574d145987600868ce1412102e" Nov 01 05:29:06.973646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:06.973612 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2322d14-f4f2-4bef-b294-1a49550b582f" path="/var/lib/kubelet/pods/c2322d14-f4f2-4bef-b294-1a49550b582f/volumes" Nov 01 05:29:08.049271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:08.049226 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz"] Nov 01 05:29:08.100020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:08.099993 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2eeea74-6hxvz"] Nov 01 05:29:08.973797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:08.973762 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a76bd4-48f3-4d0e-a452-06a05cdca804" path="/var/lib/kubelet/pods/70a76bd4-48f3-4d0e-a452-06a05cdca804/volumes" Nov 01 05:29:19.047947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:19.047870 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx"] Nov 01 05:29:19.098502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:19.098470 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd0eae6-4zgfx"] Nov 01 05:29:20.973935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:20.973902 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29d4bc-6a86-4bf7-87df-410606d60a1e" path="/var/lib/kubelet/pods/7e29d4bc-6a86-4bf7-87df-410606d60a1e/volumes" Nov 01 05:29:21.046992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:21.046950 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv"] Nov 01 05:29:21.098104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:21.098072 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dff2fe7-w59vv"] Nov 01 05:29:22.973804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:22.973762 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc67a65f-0c8e-46fb-b6cb-a1218d409bc4" path="/var/lib/kubelet/pods/fc67a65f-0c8e-46fb-b6cb-a1218d409bc4/volumes" Nov 01 05:29:26.948698 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:29:26.948660 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:29:36.048179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:36.048144 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m"] Nov 01 05:29:36.098660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:36.098626 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba67922-nl84m"] Nov 01 05:29:36.974309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:36.974275 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26428b27-6cfa-4acf-8b44-7dbb84c0e616" path="/var/lib/kubelet/pods/26428b27-6cfa-4acf-8b44-7dbb84c0e616/volumes" Nov 01 05:29:43.841845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.841807 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:29:43.842384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.842059 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" containerID="cri-o://d49cc11b901a80a619d96edafd8a3fb65c11bb465676ceee857d11f7931c151a" gracePeriod=30 Nov 01 05:29:43.893371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.893331 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:29:43.893653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.893594 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" containerID="cri-o://18633dda041990b9ada7051cd4ba2ea7b9906c5d60047214f872b9fe4578a6d0" gracePeriod=30 Nov 01 05:29:43.894655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.894635 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:29:43.894845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.894807 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" containerID="cri-o://aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722" gracePeriod=30 Nov 01 05:29:43.943808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.943777 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:29:43.979860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:43.979825 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" containerID="cri-o://7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32" gracePeriod=30 Nov 01 05:29:47.037728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:47.037691 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:29:47.038108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:47.037758 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:29:47.169534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:47.169496 4425 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:29:47.169728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:47.169559 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:29:48.048066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:48.048035 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh"] Nov 01 05:29:48.098910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:48.098877 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9170e97f-6qdvh"] Nov 01 05:29:48.974446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:48.974418 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6" path="/var/lib/kubelet/pods/36ef5dcb-be20-41a0-a3c4-ac3ac936b2c6/volumes" Nov 01 05:29:49.016465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.016439 4425 generic.go:356] "Generic (PLEG): container finished" podID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerID="18633dda041990b9ada7051cd4ba2ea7b9906c5d60047214f872b9fe4578a6d0" exitCode=0 Nov 01 05:29:49.016580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.016516 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" event={"ID":"457dd741-95f2-45ed-89f7-5a6dcafb151d","Type":"ContainerDied","Data":"18633dda041990b9ada7051cd4ba2ea7b9906c5d60047214f872b9fe4578a6d0"} Nov 01 05:29:49.018656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.018632 4425 generic.go:356] "Generic (PLEG): container finished" podID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerID="d49cc11b901a80a619d96edafd8a3fb65c11bb465676ceee857d11f7931c151a" exitCode=0 Nov 01 05:29:49.018759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.018668 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" event={"ID":"38c6daa3-cb09-4c44-ae95-582f3c6e134a","Type":"ContainerDied","Data":"d49cc11b901a80a619d96edafd8a3fb65c11bb465676ceee857d11f7931c151a"} Nov 01 05:29:49.181291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.179824 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:29:49.229871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.229843 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:29:49.249711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.249686 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:29:49.308379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.308354 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:29:49.335729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335704 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.335893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335745 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.335893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335821 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.335893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335860 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335908 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335934 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.335974 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336006 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336053 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336096 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336140 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336190 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336227 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336296 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336334 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336341 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336433 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336502 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336521 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336531 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336547 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336569 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336600 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336656 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336699 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336740 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\") pod \"457dd741-95f2-45ed-89f7-5a6dcafb151d\" (UID: \"457dd741-95f2-45ed-89f7-5a6dcafb151d\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336768 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.336846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336813 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.337540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336858 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\") pod \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\" (UID: \"38c6daa3-cb09-4c44-ae95-582f3c6e134a\") " Nov 01 05:29:49.337540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336906 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\") pod \"ec3fe719-9c8b-4773-99db-72a3027106b1\" (UID: \"ec3fe719-9c8b-4773-99db-72a3027106b1\") " Nov 01 05:29:49.337821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337579 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.337821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337602 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.337821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337616 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336811 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340597 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.336838 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337016 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337007 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337197 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.337894 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.338101 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340560 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340574 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340765 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "kube-api-access-fmdxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.340848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340772 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.341578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.340969 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.341578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.341144 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.341578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.341071 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:29:49.341578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.341324 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.343312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.343273 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.343624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.343597 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.343732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.343633 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:29:49.343732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.343676 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "kube-api-access-xpqdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.345659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.345561 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "kube-api-access-c8bcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.345659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.345613 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token" pod "38c6daa3-cb09-4c44-ae95-582f3c6e134a" (UID: "38c6daa3-cb09-4c44-ae95-582f3c6e134a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.345659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.345611 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo" pod "ec3fe719-9c8b-4773-99db-72a3027106b1" (UID: "ec3fe719-9c8b-4773-99db-72a3027106b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:29:49.345659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.345635 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo" pod "457dd741-95f2-45ed-89f7-5a6dcafb151d" (UID: "457dd741-95f2-45ed-89f7-5a6dcafb151d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:29:49.438968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.438884 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439002 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439031 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439059 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439117 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439160 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439146 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439219 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439291 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439320 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.439350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439343 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\") pod \"17e9bf74-070f-4c59-ba1f-22d2209c488e\" (UID: \"17e9bf74-070f-4c59-ba1f-22d2209c488e\") " Nov 01 05:29:49.439637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439534 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.439637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439581 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.439637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.439522 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:29:49.440072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440029 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/457dd741-95f2-45ed-89f7-5a6dcafb151d-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440067 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440085 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440101 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440167 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440268 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440310 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440325 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440339 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440354 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440367 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440382 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e9bf74-070f-4c59-ba1f-22d2209c488e-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440396 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bcg\" (UniqueName: \"kubernetes.io/projected/38c6daa3-cb09-4c44-ae95-582f3c6e134a-kube-api-access-c8bcg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440411 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440425 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440440 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440455 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440470 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe719-9c8b-4773-99db-72a3027106b1-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440486 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440500 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440516 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe719-9c8b-4773-99db-72a3027106b1-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440532 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440547 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqdn\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-kube-api-access-xpqdn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440562 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440578 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440593 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/457dd741-95f2-45ed-89f7-5a6dcafb151d-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440609 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c6daa3-cb09-4c44-ae95-582f3c6e134a-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440624 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdxq\" (UniqueName: \"kubernetes.io/projected/ec3fe719-9c8b-4773-99db-72a3027106b1-kube-api-access-fmdxq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.440912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.440639 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c6daa3-cb09-4c44-ae95-582f3c6e134a-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.441689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.441506 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:29:49.441689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.441543 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "kube-api-access-5vccj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.441797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.441781 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:29:49.441850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.441835 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy" pod "17e9bf74-070f-4c59-ba1f-22d2209c488e" (UID: "17e9bf74-070f-4c59-ba1f-22d2209c488e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:29:49.542094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.542059 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.542094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.542095 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.542380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.542111 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:49.542380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:49.542128 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vccj\" (UniqueName: \"kubernetes.io/projected/17e9bf74-070f-4c59-ba1f-22d2209c488e-kube-api-access-5vccj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:29:50.026891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.026861 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" Nov 01 05:29:50.027088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.026860 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z" event={"ID":"457dd741-95f2-45ed-89f7-5a6dcafb151d","Type":"ContainerDied","Data":"74fb3a2a74ae30b7b20ea558d3b67619519978adccf0c11e18f5c0b5f56d49d3"} Nov 01 05:29:50.027088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.027007 4425 scope.go:122] "RemoveContainer" containerID="18633dda041990b9ada7051cd4ba2ea7b9906c5d60047214f872b9fe4578a6d0" Nov 01 05:29:50.028356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.028337 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" event={"ID":"38c6daa3-cb09-4c44-ae95-582f3c6e134a","Type":"ContainerDied","Data":"131ec9ed29b5e6310f33e6624945cc8a1679d13f408705eae872c57a1f3d2485"} Nov 01 05:29:50.028443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.028342 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6" Nov 01 05:29:50.030169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.030138 4425 generic.go:356] "Generic (PLEG): container finished" podID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerID="aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722" exitCode=0 Nov 01 05:29:50.030344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.030193 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" Nov 01 05:29:50.030344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.030222 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" event={"ID":"ec3fe719-9c8b-4773-99db-72a3027106b1","Type":"ContainerDied","Data":"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722"} Nov 01 05:29:50.030344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.030310 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn" event={"ID":"ec3fe719-9c8b-4773-99db-72a3027106b1","Type":"ContainerDied","Data":"2de6b7ba2a7f71d14ebefcc3cdd98eb25349d83c0032ec0a6e3a92bcc289b042"} Nov 01 05:29:50.033006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.032982 4425 generic.go:356] "Generic (PLEG): container finished" podID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerID="7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32" exitCode=0 Nov 01 05:29:50.033148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.033038 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" Nov 01 05:29:50.033148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.033044 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" event={"ID":"17e9bf74-070f-4c59-ba1f-22d2209c488e","Type":"ContainerDied","Data":"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32"} Nov 01 05:29:50.033148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.033082 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8" event={"ID":"17e9bf74-070f-4c59-ba1f-22d2209c488e","Type":"ContainerDied","Data":"d945134f79f1d1b9492bb8c5976db064fd6f726e6509aad9c21f46aeb1cab4ce"} Nov 01 05:29:50.041052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.041030 4425 scope.go:122] "RemoveContainer" containerID="d49cc11b901a80a619d96edafd8a3fb65c11bb465676ceee857d11f7931c151a" Nov 01 05:29:50.051871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.051852 4425 scope.go:122] "RemoveContainer" containerID="aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722" Nov 01 05:29:50.061231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.061215 4425 scope.go:122] "RemoveContainer" containerID="aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722" Nov 01 05:29:50.061532 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:29:50.061513 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722\": container with ID starting with aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722 not found: ID does not exist" containerID="aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722" Nov 01 05:29:50.061574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.061540 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722"} err="failed to get container status \"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722\": rpc error: code = NotFound desc = could not find container \"aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722\": container with ID starting with aeee15eff8b8e3f6d838f425213eb2545de23bcd6e60953e15283802636db722 not found: ID does not exist" Nov 01 05:29:50.061574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.061557 4425 scope.go:122] "RemoveContainer" containerID="7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32" Nov 01 05:29:50.070309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.070289 4425 scope.go:122] "RemoveContainer" containerID="7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32" Nov 01 05:29:50.070584 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:29:50.070568 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32\": container with ID starting with 7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32 not found: ID does not exist" containerID="7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32" Nov 01 05:29:50.070634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.070592 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32"} err="failed to get container status \"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32\": rpc error: code = NotFound desc = could not find container \"7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32\": container with ID starting with 7a85faf0d3414e112336cd2ba2502b81feafa5af931f3bfc07533a9e3ca72f32 not found: ID does not exist" Nov 01 05:29:50.197303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.197196 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:29:50.249158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.249116 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-dppk8"] Nov 01 05:29:50.314922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.314868 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:29:50.319021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.318993 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-zbnn6"] Nov 01 05:29:50.431623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.431589 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:29:50.437526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.437495 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-6jj9z"] Nov 01 05:29:50.549981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.549891 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:29:50.554428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.554399 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-ck7gn"] Nov 01 05:29:50.974948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.974914 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" path="/var/lib/kubelet/pods/17e9bf74-070f-4c59-ba1f-22d2209c488e/volumes" Nov 01 05:29:50.975463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.975443 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" path="/var/lib/kubelet/pods/38c6daa3-cb09-4c44-ae95-582f3c6e134a/volumes" Nov 01 05:29:50.975904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.975886 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" path="/var/lib/kubelet/pods/457dd741-95f2-45ed-89f7-5a6dcafb151d/volumes" Nov 01 05:29:50.976639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:50.976614 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" path="/var/lib/kubelet/pods/ec3fe719-9c8b-4773-99db-72a3027106b1/volumes" Nov 01 05:29:51.900916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.900887 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:29:51.901490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901475 4425 cpu_manager.go:414] "containerMap: removing container" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901492 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901509 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901514 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901524 4425 cpu_manager.go:414] "containerMap: removing container" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901530 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901536 4425 cpu_manager.go:414] "containerMap: removing container" podUID="74d104cc-7a59-42ef-b35a-b07208246e63" containerName="revoke-keys" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901541 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d104cc-7a59-42ef-b35a-b07208246e63" containerName="revoke-keys" Nov 01 05:29:51.901564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901565 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901570 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901579 4425 cpu_manager.go:414] "containerMap: removing container" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901583 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901675 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="457dd741-95f2-45ed-89f7-5a6dcafb151d" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901684 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d104cc-7a59-42ef-b35a-b07208246e63" containerName="revoke-keys" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901693 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901700 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3fe719-9c8b-4773-99db-72a3027106b1" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901707 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e9bf74-070f-4c59-ba1f-22d2209c488e" containerName="istio-proxy" Nov 01 05:29:51.901823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.901715 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c6daa3-cb09-4c44-ae95-582f3c6e134a" containerName="istio-proxy" Nov 01 05:29:51.906976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.906958 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:51.961031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:51.961000 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\"" Nov 01 05:29:52.015967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.015930 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:29:52.075885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.075831 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.075989 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076084 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076128 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496hp\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076211 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076276 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076320 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076469 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.076552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.076515 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178104 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178155 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178178 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178229 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178248 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178368 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178419 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178491 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178520 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496hp\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178631 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178706 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178780 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.178932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.178895 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.179039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.179015 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.180782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.180758 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.180931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.180912 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.241000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.240968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.241160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.241101 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-496hp\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.520123 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.520038 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:52.952839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.952815 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:29:52.954197 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:29:52.954171 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc403328e_3f76_4ba8_a5d6_a1a4cd810494.slice/crio-70ed9bf355d0597dd4559297ff29704e2abcb993a93b2217421660a186fb1709 WatchSource:0}: Error finding container 70ed9bf355d0597dd4559297ff29704e2abcb993a93b2217421660a186fb1709: Status 404 returned error can't find the container with id 70ed9bf355d0597dd4559297ff29704e2abcb993a93b2217421660a186fb1709 Nov 01 05:29:52.956768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.956732 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:29:52.956873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.956799 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:29:52.956873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:52.956827 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:29:53.060233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:53.060195 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" event={"ID":"c403328e-3f76-4ba8-a5d6-a1a4cd810494","Type":"ContainerStarted","Data":"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286"} Nov 01 05:29:53.060429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:53.060270 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" event={"ID":"c403328e-3f76-4ba8-a5d6-a1a4cd810494","Type":"ContainerStarted","Data":"70ed9bf355d0597dd4559297ff29704e2abcb993a93b2217421660a186fb1709"} Nov 01 05:29:53.187265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:53.187199 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podStartSLOduration=2.187177441 podStartE2EDuration="2.187177441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:29:53.182948295 +0000 UTC m=+1608.323537238" watchObservedRunningTime="2026-11-01 05:29:53.187177441 +0000 UTC m=+1608.327766359" Nov 01 05:29:53.520153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:53.520124 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:29:54.522930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:54.522899 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:54.523322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:54.522956 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:29:55.520550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:55.520517 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:55.520833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:55.520799 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:29:56.520692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:56.520649 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:56.521192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:56.520714 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:29:56.975185 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:29:56.975143 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:29:57.520659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:57.520628 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:57.520854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:57.520684 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:29:58.520583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:58.520552 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:58.520906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:58.520624 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:29:59.520379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:59.520341 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 05:29:59.520561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:29:59.520395 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 05:30:00.163947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.163906 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:00.164843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.164819 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:30:00.164931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.164846 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:30:00.165077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.165062 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa71ec23-30b9-460c-b9b6-e054d0640203" containerName="istio-proxy" Nov 01 05:30:00.169216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.169190 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:00.169372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.169345 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.174140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.174119 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:00.174310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.174280 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.178632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.178609 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.196297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.196266 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.196474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.196453 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ztf\" (UniqueName: \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.196588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.196568 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmd7\" (UniqueName: \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.196777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.196760 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.215369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.215343 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:00.219424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.219402 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:00.219546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.219523 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.223656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.223633 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:00.223759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.223714 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.223759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.223731 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:30:00.223861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.223847 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:30:00.228860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.228507 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:00.228860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.228670 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.235127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.235101 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.263681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.263654 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:30:00.267234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.267209 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:00.270462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.270435 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:00.274269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.274228 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:00.276873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.276848 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:00.297871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.297836 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9wk\" (UniqueName: \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.298069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.297933 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.298069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.297992 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.298069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298022 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.298232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298089 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.298232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298152 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.298232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298193 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.298445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298335 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.298445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298408 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.298545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298447 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8d5\" (UniqueName: \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.298545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298474 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvrr\" (UniqueName: \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.298545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298492 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46lc\" (UniqueName: \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.298682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298545 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ztf\" (UniqueName: \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.298682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298566 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmd7\" (UniqueName: \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.298860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.298841 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.299423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.299408 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.324722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.324686 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:00.329681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.329649 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:00.361446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.361418 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmd7\" (UniqueName: \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.361624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.361418 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ztf\" (UniqueName: \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.371340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.371313 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:00.399898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.399859 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9wk\" (UniqueName: \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.399898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.399900 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.400137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.399954 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.400137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.399986 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.400137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400021 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.400137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400058 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.400419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400170 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.400419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400243 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8d5\" (UniqueName: \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.400419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400318 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvrr\" (UniqueName: \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.400419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400344 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46lc\" (UniqueName: \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.401399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400889 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.401399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400888 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.401399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.400995 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.401399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.401338 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.402190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.402139 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.463125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.463053 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.463125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.463070 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvrr\" (UniqueName: \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\") pod \"maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.463332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.463190 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8d5\" (UniqueName: \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.463504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.463483 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46lc\" (UniqueName: \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.463539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.463483 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9wk\" (UniqueName: \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.488109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.488083 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:00.499774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.499747 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:00.506843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.506775 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:00.528392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.528360 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:30:00.529575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.529555 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:30:00.530337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.530316 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:30:00.538321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.538294 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:00.548349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.548313 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:00.557784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.557751 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:00.567632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.566553 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:00.887297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.887232 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:00.889047 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:00.889021 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd47d121_6d91_4e9d_b497_586dc931ce80.slice/crio-de2d2c5947cb893f02f7730cb178ee25c294fe5c61bfea8cf110ac416f5860b5 WatchSource:0}: Error finding container de2d2c5947cb893f02f7730cb178ee25c294fe5c61bfea8cf110ac416f5860b5: Status 404 returned error can't find the container with id de2d2c5947cb893f02f7730cb178ee25c294fe5c61bfea8cf110ac416f5860b5 Nov 01 05:30:00.941620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.941593 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:00.942505 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:00.942482 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11915e9_cafa_4c38_bfd6_c08b2d672fc9.slice/crio-b6fd69ce0bef3dfb81260eca7459664e1cdf6e81b0277993509f317729335b63 WatchSource:0}: Error finding container b6fd69ce0bef3dfb81260eca7459664e1cdf6e81b0277993509f317729335b63: Status 404 returned error can't find the container with id b6fd69ce0bef3dfb81260eca7459664e1cdf6e81b0277993509f317729335b63 Nov 01 05:30:00.991822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.991793 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:00.991962 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:00.991898 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c981ce_7b33_4ec9_a9c0_fff070e31929.slice/crio-62b9e9f719c2dbe38bbb34f17c31f3864dc4fe1ed1793c45b35c55710b81f38a WatchSource:0}: Error finding container 62b9e9f719c2dbe38bbb34f17c31f3864dc4fe1ed1793c45b35c55710b81f38a: Status 404 returned error can't find the container with id 62b9e9f719c2dbe38bbb34f17c31f3864dc4fe1ed1793c45b35c55710b81f38a Nov 01 05:30:00.992707 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:00.992684 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69493047_8928_41c8_ae8c_12238aad3dd9.slice/crio-a53cf08159a63208fdf89963a59c7c63acbc549050aac420040fcf53039b9fc2 WatchSource:0}: Error finding container a53cf08159a63208fdf89963a59c7c63acbc549050aac420040fcf53039b9fc2: Status 404 returned error can't find the container with id a53cf08159a63208fdf89963a59c7c63acbc549050aac420040fcf53039b9fc2 Nov 01 05:30:00.995092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:00.995070 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:01.051340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.051310 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:01.054845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.054823 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:01.122089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.122041 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerStarted","Data":"62b9e9f719c2dbe38bbb34f17c31f3864dc4fe1ed1793c45b35c55710b81f38a"} Nov 01 05:30:01.124162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.124133 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerStarted","Data":"de2d2c5947cb893f02f7730cb178ee25c294fe5c61bfea8cf110ac416f5860b5"} Nov 01 05:30:01.125822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.125773 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerStarted","Data":"a53cf08159a63208fdf89963a59c7c63acbc549050aac420040fcf53039b9fc2"} Nov 01 05:30:01.127670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.127647 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerStarted","Data":"b6fd69ce0bef3dfb81260eca7459664e1cdf6e81b0277993509f317729335b63"} Nov 01 05:30:01.149284 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:01.149225 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13c484c_ea80_434e_b773_dcb061d70eb1.slice/crio-9c94bd7b1efd2dde346b2be1e148f01685ab69dd2d357c7d44a5ba7cd9545388 WatchSource:0}: Error finding container 9c94bd7b1efd2dde346b2be1e148f01685ab69dd2d357c7d44a5ba7cd9545388: Status 404 returned error can't find the container with id 9c94bd7b1efd2dde346b2be1e148f01685ab69dd2d357c7d44a5ba7cd9545388 Nov 01 05:30:01.154141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:01.154115 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:02.136298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.136245 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerStarted","Data":"a941885b078e6a10d8bf66a62a115cbfe129cda2f1d499b52e63a358a4e6d0fa"} Nov 01 05:30:02.136667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.136354 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerStarted","Data":"cf17ad2b6a4772831f6ae74e16218a2f0a332dec226a054279b1dd31f28bdcc4"} Nov 01 05:30:02.137914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.137891 4425 generic.go:356] "Generic (PLEG): container finished" podID="69493047-8928-41c8-ae8c-12238aad3dd9" containerID="846fe9e5744d2090d98765392953b04f476f0285e5f745af2613bcf89a2d7b32" exitCode=60 Nov 01 05:30:02.138050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.137955 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerDied","Data":"846fe9e5744d2090d98765392953b04f476f0285e5f745af2613bcf89a2d7b32"} Nov 01 05:30:02.138193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.138176 4425 scope.go:122] "RemoveContainer" containerID="846fe9e5744d2090d98765392953b04f476f0285e5f745af2613bcf89a2d7b32" Nov 01 05:30:02.139682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.139659 4425 generic.go:356] "Generic (PLEG): container finished" podID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerID="0a384bc0dc34038d85cc229bd940685d3eaaed40923ed3d6c143f41e17f0c0e1" exitCode=60 Nov 01 05:30:02.139862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.139731 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerDied","Data":"0a384bc0dc34038d85cc229bd940685d3eaaed40923ed3d6c143f41e17f0c0e1"} Nov 01 05:30:02.139917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.139909 4425 scope.go:122] "RemoveContainer" containerID="0a384bc0dc34038d85cc229bd940685d3eaaed40923ed3d6c143f41e17f0c0e1" Nov 01 05:30:02.141723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.141343 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerStarted","Data":"805c052f14386e43b47d85f1403d4bd723fa690a6afaa81d034ffdd58be94902"} Nov 01 05:30:02.141723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.141382 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerStarted","Data":"9c94bd7b1efd2dde346b2be1e148f01685ab69dd2d357c7d44a5ba7cd9545388"} Nov 01 05:30:02.144464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.144442 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerStarted","Data":"7ecc27feb1c0c5dab57b21d400ad16bfe54d0a5f21b9894dc131402dc24d8198"} Nov 01 05:30:02.146194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.146171 4425 generic.go:356] "Generic (PLEG): container finished" podID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerID="4a780d4bf1210baa0a06cb3caafbe2ed579bd836f19508c84b81834299ec37fa" exitCode=60 Nov 01 05:30:02.146644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.146269 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerDied","Data":"4a780d4bf1210baa0a06cb3caafbe2ed579bd836f19508c84b81834299ec37fa"} Nov 01 05:30:02.146644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.146489 4425 scope.go:122] "RemoveContainer" containerID="4a780d4bf1210baa0a06cb3caafbe2ed579bd836f19508c84b81834299ec37fa" Nov 01 05:30:02.148113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.148091 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerStarted","Data":"3f4ca2f50dd4c7d38ea3490605c8d84ef3035c0b120e15f97980f285c4922581"} Nov 01 05:30:02.148216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.148132 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerStarted","Data":"735380377b26fe428ccaad7b26144acf542892b76ee4c36a9136bc2f65c9d54b"} Nov 01 05:30:02.259328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.259245 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" podStartSLOduration=2.259232574 podStartE2EDuration="2.259232574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:02.256583937 +0000 UTC m=+1617.397172880" watchObservedRunningTime="2026-11-01 05:30:02.259232574 +0000 UTC m=+1617.399821518" Nov 01 05:30:02.492684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.492632 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" podStartSLOduration=2.492616649 podStartE2EDuration="2.492616649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:02.491369918 +0000 UTC m=+1617.631958838" watchObservedRunningTime="2026-11-01 05:30:02.492616649 +0000 UTC m=+1617.633205596" Nov 01 05:30:02.844989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.844933 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" podStartSLOduration=2.8449183529999997 podStartE2EDuration="2.844918353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:02.842664826 +0000 UTC m=+1617.983253743" watchObservedRunningTime="2026-11-01 05:30:02.844918353 +0000 UTC m=+1617.985507271" Nov 01 05:30:02.962902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:02.962782 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" podStartSLOduration=2.9627658390000002 podStartE2EDuration="2.962765839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:02.962137077 +0000 UTC m=+1618.102725996" watchObservedRunningTime="2026-11-01 05:30:02.962765839 +0000 UTC m=+1618.103354757" Nov 01 05:30:03.157096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.157064 4425 generic.go:356] "Generic (PLEG): container finished" podID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerID="805c052f14386e43b47d85f1403d4bd723fa690a6afaa81d034ffdd58be94902" exitCode=60 Nov 01 05:30:03.157602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.157150 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerDied","Data":"805c052f14386e43b47d85f1403d4bd723fa690a6afaa81d034ffdd58be94902"} Nov 01 05:30:03.157602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.157463 4425 scope.go:122] "RemoveContainer" containerID="805c052f14386e43b47d85f1403d4bd723fa690a6afaa81d034ffdd58be94902" Nov 01 05:30:03.158968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.158947 4425 generic.go:356] "Generic (PLEG): container finished" podID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerID="7ecc27feb1c0c5dab57b21d400ad16bfe54d0a5f21b9894dc131402dc24d8198" exitCode=60 Nov 01 05:30:03.159078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.159030 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerDied","Data":"7ecc27feb1c0c5dab57b21d400ad16bfe54d0a5f21b9894dc131402dc24d8198"} Nov 01 05:30:03.159301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.159286 4425 scope.go:122] "RemoveContainer" containerID="7ecc27feb1c0c5dab57b21d400ad16bfe54d0a5f21b9894dc131402dc24d8198" Nov 01 05:30:03.162044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.162021 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerStarted","Data":"ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359"} Nov 01 05:30:03.163690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.163663 4425 generic.go:356] "Generic (PLEG): container finished" podID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerID="3f4ca2f50dd4c7d38ea3490605c8d84ef3035c0b120e15f97980f285c4922581" exitCode=60 Nov 01 05:30:03.163790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.163739 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerDied","Data":"3f4ca2f50dd4c7d38ea3490605c8d84ef3035c0b120e15f97980f285c4922581"} Nov 01 05:30:03.163924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.163909 4425 scope.go:122] "RemoveContainer" containerID="3f4ca2f50dd4c7d38ea3490605c8d84ef3035c0b120e15f97980f285c4922581" Nov 01 05:30:03.169758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.169738 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerID="a941885b078e6a10d8bf66a62a115cbfe129cda2f1d499b52e63a358a4e6d0fa" exitCode=60 Nov 01 05:30:03.169886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.169826 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerDied","Data":"a941885b078e6a10d8bf66a62a115cbfe129cda2f1d499b52e63a358a4e6d0fa"} Nov 01 05:30:03.170044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.170029 4425 scope.go:122] "RemoveContainer" containerID="a941885b078e6a10d8bf66a62a115cbfe129cda2f1d499b52e63a358a4e6d0fa" Nov 01 05:30:03.173463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.173435 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerStarted","Data":"c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103"} Nov 01 05:30:03.175473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.175454 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerStarted","Data":"91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed"} Nov 01 05:30:03.392840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.392792 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" podStartSLOduration=3.392778931 podStartE2EDuration="3.392778931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:03.391431343 +0000 UTC m=+1618.532020274" watchObservedRunningTime="2026-11-01 05:30:03.392778931 +0000 UTC m=+1618.533367848" Nov 01 05:30:03.510950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.510865 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" podStartSLOduration=3.510850615 podStartE2EDuration="3.510850615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:03.509067943 +0000 UTC m=+1618.649656885" watchObservedRunningTime="2026-11-01 05:30:03.510850615 +0000 UTC m=+1618.651439523" Nov 01 05:30:03.980764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:03.980710 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" podStartSLOduration=3.980697485 podStartE2EDuration="3.980697485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:03.978524468 +0000 UTC m=+1619.119113385" watchObservedRunningTime="2026-11-01 05:30:03.980697485 +0000 UTC m=+1619.121286402" Nov 01 05:30:04.184923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.184822 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerStarted","Data":"e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1"} Nov 01 05:30:04.186870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.186844 4425 generic.go:356] "Generic (PLEG): container finished" podID="69493047-8928-41c8-ae8c-12238aad3dd9" containerID="c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103" exitCode=60 Nov 01 05:30:04.186999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.186922 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerDied","Data":"c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103"} Nov 01 05:30:04.186999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.186979 4425 scope.go:122] "RemoveContainer" containerID="846fe9e5744d2090d98765392953b04f476f0285e5f745af2613bcf89a2d7b32" Nov 01 05:30:04.187197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.187180 4425 scope.go:122] "RemoveContainer" containerID="c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103" Nov 01 05:30:04.187536 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:04.187496 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8_odh-ai-gateway-infra(69493047-8928-41c8-ae8c-12238aad3dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" Nov 01 05:30:04.188860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.188842 4425 generic.go:356] "Generic (PLEG): container finished" podID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerID="91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed" exitCode=60 Nov 01 05:30:04.188978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.188927 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerDied","Data":"91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed"} Nov 01 05:30:04.189142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.189120 4425 scope.go:122] "RemoveContainer" containerID="91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed" Nov 01 05:30:04.189464 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:04.189443 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s_odh-ai-gateway-infra(f11915e9-cafa-4c38-bfd6-c08b2d672fc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" Nov 01 05:30:04.190886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.190862 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerStarted","Data":"aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52"} Nov 01 05:30:04.192662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.192641 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerStarted","Data":"ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3"} Nov 01 05:30:04.194449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.194430 4425 generic.go:356] "Generic (PLEG): container finished" podID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerID="ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359" exitCode=60 Nov 01 05:30:04.194544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.194495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerDied","Data":"ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359"} Nov 01 05:30:04.194740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.194723 4425 scope.go:122] "RemoveContainer" containerID="ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359" Nov 01 05:30:04.194998 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:04.194972 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w_odh-ai-gateway-infra(cd47d121-6d91-4e9d-b497-586dc931ce80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" Nov 01 05:30:04.196388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.196369 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerStarted","Data":"bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb"} Nov 01 05:30:04.201018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.200999 4425 scope.go:122] "RemoveContainer" containerID="0a384bc0dc34038d85cc229bd940685d3eaaed40923ed3d6c143f41e17f0c0e1" Nov 01 05:30:04.214324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:04.214308 4425 scope.go:122] "RemoveContainer" containerID="4a780d4bf1210baa0a06cb3caafbe2ed579bd836f19508c84b81834299ec37fa" Nov 01 05:30:05.205600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.205565 4425 scope.go:122] "RemoveContainer" containerID="ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359" Nov 01 05:30:05.206019 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.205911 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w_odh-ai-gateway-infra(cd47d121-6d91-4e9d-b497-586dc931ce80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" Nov 01 05:30:05.207133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.207109 4425 generic.go:356] "Generic (PLEG): container finished" podID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerID="bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb" exitCode=60 Nov 01 05:30:05.207290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.207242 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerDied","Data":"bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb"} Nov 01 05:30:05.207353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.207337 4425 scope.go:122] "RemoveContainer" containerID="3f4ca2f50dd4c7d38ea3490605c8d84ef3035c0b120e15f97980f285c4922581" Nov 01 05:30:05.207517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.207498 4425 scope.go:122] "RemoveContainer" containerID="bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb" Nov 01 05:30:05.207809 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.207787 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn_odh-ai-gateway-infra(ad72f17a-1ccf-47c8-b80b-6f733afaf1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" Nov 01 05:30:05.209157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.209124 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerID="e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1" exitCode=60 Nov 01 05:30:05.209292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.209181 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerDied","Data":"e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1"} Nov 01 05:30:05.209511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.209498 4425 scope.go:122] "RemoveContainer" containerID="e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1" Nov 01 05:30:05.209779 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.209758 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2_odh-ai-gateway-infra(9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" Nov 01 05:30:05.211063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.211044 4425 scope.go:122] "RemoveContainer" containerID="c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103" Nov 01 05:30:05.211351 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.211320 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8_odh-ai-gateway-infra(69493047-8928-41c8-ae8c-12238aad3dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" Nov 01 05:30:05.212763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.212740 4425 scope.go:122] "RemoveContainer" containerID="91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed" Nov 01 05:30:05.212983 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.212966 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s_odh-ai-gateway-infra(f11915e9-cafa-4c38-bfd6-c08b2d672fc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" Nov 01 05:30:05.214440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.214418 4425 generic.go:356] "Generic (PLEG): container finished" podID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerID="aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52" exitCode=60 Nov 01 05:30:05.214523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.214494 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerDied","Data":"aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52"} Nov 01 05:30:05.214683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.214663 4425 scope.go:122] "RemoveContainer" containerID="aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52" Nov 01 05:30:05.214919 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.214898 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8_odh-ai-gateway-infra(b13c484c-ea80-434e-b773-dcb061d70eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" Nov 01 05:30:05.216607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.216583 4425 generic.go:356] "Generic (PLEG): container finished" podID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerID="ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3" exitCode=60 Nov 01 05:30:05.216686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.216609 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerDied","Data":"ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3"} Nov 01 05:30:05.216807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.216794 4425 scope.go:122] "RemoveContainer" containerID="ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3" Nov 01 05:30:05.217014 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:05.216997 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5_odh-ai-gateway-infra(c1c981ce-7b33-4ec9-a9c0-fff070e31929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" Nov 01 05:30:05.221156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.221137 4425 scope.go:122] "RemoveContainer" containerID="a941885b078e6a10d8bf66a62a115cbfe129cda2f1d499b52e63a358a4e6d0fa" Nov 01 05:30:05.234233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.234212 4425 scope.go:122] "RemoveContainer" containerID="805c052f14386e43b47d85f1403d4bd723fa690a6afaa81d034ffdd58be94902" Nov 01 05:30:05.247177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:05.247160 4425 scope.go:122] "RemoveContainer" containerID="7ecc27feb1c0c5dab57b21d400ad16bfe54d0a5f21b9894dc131402dc24d8198" Nov 01 05:30:06.150757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.150727 4425 scope.go:122] "RemoveContainer" containerID="1413f30a01380749f6902ef33f510131a232f04c657827dd76c6b01075d34c6d" Nov 01 05:30:06.160740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.160720 4425 scope.go:122] "RemoveContainer" containerID="d254099c34bbc210b985c10cbfd05931e194faa1fefd61654d8ed22198d505b8" Nov 01 05:30:06.169672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.169655 4425 scope.go:122] "RemoveContainer" containerID="852b611a864b5f079deb69d8712a25671f70f60bb11178e67b917f8d506bf3a1" Nov 01 05:30:06.179310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.179293 4425 scope.go:122] "RemoveContainer" containerID="e55872acb3de70f45fb96b0a5e6c6abc92e60cef64fb1a659eb3d77387d32bc3" Nov 01 05:30:06.188900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.188876 4425 scope.go:122] "RemoveContainer" containerID="f42e4e648b0bd7fe7dad724cc72772aace774fa036eb36e5a8e5b824ce25d8f2" Nov 01 05:30:06.201075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.201052 4425 scope.go:122] "RemoveContainer" containerID="fa0d2f2084f202f64fd9766c4278c8cd1a82b8cb358e4f6a30bfcf2f825e6efb" Nov 01 05:30:06.210191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.210017 4425 scope.go:122] "RemoveContainer" containerID="2a262b875850c3298303f89805d4e387ccee337c24ddef9716436c9c4a03c994" Nov 01 05:30:06.218655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.218639 4425 scope.go:122] "RemoveContainer" containerID="3d51e4076a7c8c705766d7990ab8edfa7b60514482d6d22bebfba6073f7fbdf8" Nov 01 05:30:06.226369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.226245 4425 scope.go:122] "RemoveContainer" containerID="aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52" Nov 01 05:30:06.226735 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:06.226706 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8_odh-ai-gateway-infra(b13c484c-ea80-434e-b773-dcb061d70eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" Nov 01 05:30:06.229000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.228979 4425 scope.go:122] "RemoveContainer" containerID="ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3" Nov 01 05:30:06.229604 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:06.229582 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5_odh-ai-gateway-infra(c1c981ce-7b33-4ec9-a9c0-fff070e31929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" Nov 01 05:30:06.230743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.230724 4425 scope.go:122] "RemoveContainer" containerID="bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb" Nov 01 05:30:06.230970 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:06.230950 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn_odh-ai-gateway-infra(ad72f17a-1ccf-47c8-b80b-6f733afaf1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" Nov 01 05:30:06.235718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:06.235699 4425 scope.go:122] "RemoveContainer" containerID="e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1" Nov 01 05:30:06.235974 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:06.235951 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2_odh-ai-gateway-infra(9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" Nov 01 05:30:11.050224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:11.050178 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt"] Nov 01 05:30:11.102407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:11.102373 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbd670d-p2zbt"] Nov 01 05:30:12.974593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:12.974555 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d219a5-ab5f-4838-8526-6db70115ee2e" path="/var/lib/kubelet/pods/20d219a5-ab5f-4838-8526-6db70115ee2e/volumes" Nov 01 05:30:15.966675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:15.966644 4425 scope.go:122] "RemoveContainer" containerID="aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52" Nov 01 05:30:15.967151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:15.966781 4425 scope.go:122] "RemoveContainer" containerID="c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103" Nov 01 05:30:15.967151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:15.966929 4425 scope.go:122] "RemoveContainer" containerID="bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb" Nov 01 05:30:15.967151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:15.967033 4425 scope.go:122] "RemoveContainer" containerID="e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1" Nov 01 05:30:16.966694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:16.966655 4425 scope.go:122] "RemoveContainer" containerID="91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed" Nov 01 05:30:17.309615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.309587 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerID="40b388e383f0655e6fa604bedfd60e85c5bcb71c4cc23f6eb060d4d668393f0a" exitCode=60 Nov 01 05:30:17.309761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.309666 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerDied","Data":"40b388e383f0655e6fa604bedfd60e85c5bcb71c4cc23f6eb060d4d668393f0a"} Nov 01 05:30:17.309761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.309727 4425 scope.go:122] "RemoveContainer" containerID="e1d000c91875750e394be31ac65acb5d4f259a8a65cf772feecf81f9704dd0b1" Nov 01 05:30:17.310141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.310124 4425 scope.go:122] "RemoveContainer" containerID="40b388e383f0655e6fa604bedfd60e85c5bcb71c4cc23f6eb060d4d668393f0a" Nov 01 05:30:17.310496 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:17.310470 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2_odh-ai-gateway-infra(9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" Nov 01 05:30:17.311743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.311720 4425 generic.go:356] "Generic (PLEG): container finished" podID="69493047-8928-41c8-ae8c-12238aad3dd9" containerID="22176c352ce79d2601520409227322e0fe73f86b9bd140c2626178638643453a" exitCode=60 Nov 01 05:30:17.311817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.311748 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerDied","Data":"22176c352ce79d2601520409227322e0fe73f86b9bd140c2626178638643453a"} Nov 01 05:30:17.312153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.312136 4425 scope.go:122] "RemoveContainer" containerID="22176c352ce79d2601520409227322e0fe73f86b9bd140c2626178638643453a" Nov 01 05:30:17.312478 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:17.312444 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8_odh-ai-gateway-infra(69493047-8928-41c8-ae8c-12238aad3dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" Nov 01 05:30:17.314011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.313990 4425 generic.go:356] "Generic (PLEG): container finished" podID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerID="ba8e88d132a402bb0b1fc5ac27f2ddfe27475f66c820841f00b058463cce8eed" exitCode=60 Nov 01 05:30:17.314106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.314056 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerDied","Data":"ba8e88d132a402bb0b1fc5ac27f2ddfe27475f66c820841f00b058463cce8eed"} Nov 01 05:30:17.314398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.314383 4425 scope.go:122] "RemoveContainer" containerID="ba8e88d132a402bb0b1fc5ac27f2ddfe27475f66c820841f00b058463cce8eed" Nov 01 05:30:17.314650 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:17.314630 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8_odh-ai-gateway-infra(b13c484c-ea80-434e-b773-dcb061d70eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" Nov 01 05:30:17.316351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.316333 4425 generic.go:356] "Generic (PLEG): container finished" podID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerID="4a5c267716cb7d8afc1754694352625011853d99eb87db59c2ccdac77700e1d9" exitCode=60 Nov 01 05:30:17.316449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.316379 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerDied","Data":"4a5c267716cb7d8afc1754694352625011853d99eb87db59c2ccdac77700e1d9"} Nov 01 05:30:17.316630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.316619 4425 scope.go:122] "RemoveContainer" containerID="4a5c267716cb7d8afc1754694352625011853d99eb87db59c2ccdac77700e1d9" Nov 01 05:30:17.316836 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:17.316810 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn_odh-ai-gateway-infra(ad72f17a-1ccf-47c8-b80b-6f733afaf1fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" Nov 01 05:30:17.325709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.325692 4425 scope.go:122] "RemoveContainer" containerID="c9acad2178a17d3c090db25b244f903795b145585aeb1c2f98839ae310488103" Nov 01 05:30:17.338865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.338842 4425 scope.go:122] "RemoveContainer" containerID="aabf97b99287a43abf232237b63ab279e4988638359e367762c3b13b3050ba52" Nov 01 05:30:17.354188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.354169 4425 scope.go:122] "RemoveContainer" containerID="bc38dfa2b34afa8ffbd910020b6f83c4718a93aaefb6a1c75e32125ff9b8bafb" Nov 01 05:30:17.966663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:17.966633 4425 scope.go:122] "RemoveContainer" containerID="ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359" Nov 01 05:30:18.328063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.328036 4425 generic.go:356] "Generic (PLEG): container finished" podID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerID="ee8e6c2f86b4b823bf86e64654bbbb96a642e69253dc386059cb84126c8bc7f3" exitCode=60 Nov 01 05:30:18.328568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.328105 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerDied","Data":"ee8e6c2f86b4b823bf86e64654bbbb96a642e69253dc386059cb84126c8bc7f3"} Nov 01 05:30:18.328568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.328164 4425 scope.go:122] "RemoveContainer" containerID="91deef06238eb422ef4dff847402ce86c3a3df087602c934f4a453770f0e5bed" Nov 01 05:30:18.328674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.328606 4425 scope.go:122] "RemoveContainer" containerID="ee8e6c2f86b4b823bf86e64654bbbb96a642e69253dc386059cb84126c8bc7f3" Nov 01 05:30:18.328981 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:18.328953 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s_odh-ai-gateway-infra(f11915e9-cafa-4c38-bfd6-c08b2d672fc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" Nov 01 05:30:18.331686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.331657 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerStarted","Data":"953570386f9ad245a14404136866458d29e7a51c62033f863756e8054b174fb6"} Nov 01 05:30:18.438384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.438307 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:18.557325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.557296 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:18.617642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.617580 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:18.674701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.674671 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:18.739532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.739502 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\") pod \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " Nov 01 05:30:18.739652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.739635 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\" (UniqueName: \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\") pod \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\" (UID: \"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7\") " Nov 01 05:30:18.739953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.739926 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca" pod "9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" (UID: "9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:18.740150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.740133 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:18.742371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.742347 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr" pod "9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" (UID: "9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7"). InnerVolumeSpecName "kube-api-access-xhvrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:18.778957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.778931 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:18.793848 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.793817 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:18.841528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.841501 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhvrr\" (UniqueName: \"kubernetes.io/projected/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7-kube-api-access-xhvrr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:18.841804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.841781 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:18.942415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.942379 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\") pod \"69493047-8928-41c8-ae8c-12238aad3dd9\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " Nov 01 05:30:18.942576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.942438 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\" (UniqueName: \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\") pod \"69493047-8928-41c8-ae8c-12238aad3dd9\" (UID: \"69493047-8928-41c8-ae8c-12238aad3dd9\") " Nov 01 05:30:18.942650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.942614 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\") pod \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " Nov 01 05:30:18.942702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.942664 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\" (UniqueName: \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\") pod \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\" (UID: \"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe\") " Nov 01 05:30:18.943119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.943089 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca" pod "69493047-8928-41c8-ae8c-12238aad3dd9" (UID: "69493047-8928-41c8-ae8c-12238aad3dd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:18.943421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.943364 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca" pod "ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" (UID: "ad72f17a-1ccf-47c8-b80b-6f733afaf1fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:18.944951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.944928 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk" pod "69493047-8928-41c8-ae8c-12238aad3dd9" (UID: "69493047-8928-41c8-ae8c-12238aad3dd9"). InnerVolumeSpecName "kube-api-access-7n9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:18.945641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.945618 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc" pod "ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" (UID: "ad72f17a-1ccf-47c8-b80b-6f733afaf1fe"). InnerVolumeSpecName "kube-api-access-w46lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:18.959828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.959805 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:18.966570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:18.966548 4425 scope.go:122] "RemoveContainer" containerID="ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3" Nov 01 05:30:19.048970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.044482 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69493047-8928-41c8-ae8c-12238aad3dd9-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.048970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.044518 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n9wk\" (UniqueName: \"kubernetes.io/projected/69493047-8928-41c8-ae8c-12238aad3dd9-kube-api-access-7n9wk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.048970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.044537 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.048970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.044561 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w46lc\" (UniqueName: \"kubernetes.io/projected/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe-kube-api-access-w46lc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.145152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.145111 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\" (UniqueName: \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\") pod \"b13c484c-ea80-434e-b773-dcb061d70eb1\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " Nov 01 05:30:19.145376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.145216 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\") pod \"b13c484c-ea80-434e-b773-dcb061d70eb1\" (UID: \"b13c484c-ea80-434e-b773-dcb061d70eb1\") " Nov 01 05:30:19.145637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.145604 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca" pod "b13c484c-ea80-434e-b773-dcb061d70eb1" (UID: "b13c484c-ea80-434e-b773-dcb061d70eb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:19.145870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.145829 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b13c484c-ea80-434e-b773-dcb061d70eb1-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.147662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.147640 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5" pod "b13c484c-ea80-434e-b773-dcb061d70eb1" (UID: "b13c484c-ea80-434e-b773-dcb061d70eb1"). InnerVolumeSpecName "kube-api-access-qn8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:19.247135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.247103 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8d5\" (UniqueName: \"kubernetes.io/projected/b13c484c-ea80-434e-b773-dcb061d70eb1-kube-api-access-qn8d5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.342030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.341994 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerStarted","Data":"880a81498833823e9b2c5836c2057619fa356f426177fd123ab79c71dfb8dea3"} Nov 01 05:30:19.343916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.343887 4425 generic.go:356] "Generic (PLEG): container finished" podID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerID="953570386f9ad245a14404136866458d29e7a51c62033f863756e8054b174fb6" exitCode=60 Nov 01 05:30:19.344040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.343953 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerDied","Data":"953570386f9ad245a14404136866458d29e7a51c62033f863756e8054b174fb6"} Nov 01 05:30:19.344040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.344007 4425 scope.go:122] "RemoveContainer" containerID="ec3da327ce289b97bf2b9180c0cc4f2681c076d7071eb4fb1475c75e910da359" Nov 01 05:30:19.344421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.344402 4425 scope.go:122] "RemoveContainer" containerID="953570386f9ad245a14404136866458d29e7a51c62033f863756e8054b174fb6" Nov 01 05:30:19.344756 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:19.344730 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w_odh-ai-gateway-infra(cd47d121-6d91-4e9d-b497-586dc931ce80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" Nov 01 05:30:19.345579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.345564 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" Nov 01 05:30:19.345662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.345575 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn" event={"ID":"ad72f17a-1ccf-47c8-b80b-6f733afaf1fe","Type":"ContainerDied","Data":"735380377b26fe428ccaad7b26144acf542892b76ee4c36a9136bc2f65c9d54b"} Nov 01 05:30:19.347144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.347125 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" Nov 01 05:30:19.347284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.347133 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2" event={"ID":"9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7","Type":"ContainerDied","Data":"cf17ad2b6a4772831f6ae74e16218a2f0a332dec226a054279b1dd31f28bdcc4"} Nov 01 05:30:19.348891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.348873 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" Nov 01 05:30:19.348985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.348890 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8" event={"ID":"69493047-8928-41c8-ae8c-12238aad3dd9","Type":"ContainerDied","Data":"a53cf08159a63208fdf89963a59c7c63acbc549050aac420040fcf53039b9fc2"} Nov 01 05:30:19.352302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.352282 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" event={"ID":"b13c484c-ea80-434e-b773-dcb061d70eb1","Type":"ContainerDied","Data":"9c94bd7b1efd2dde346b2be1e148f01685ab69dd2d357c7d44a5ba7cd9545388"} Nov 01 05:30:19.352396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.352322 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8" Nov 01 05:30:19.357841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.357823 4425 scope.go:122] "RemoveContainer" containerID="4a5c267716cb7d8afc1754694352625011853d99eb87db59c2ccdac77700e1d9" Nov 01 05:30:19.366789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.366768 4425 scope.go:122] "RemoveContainer" containerID="40b388e383f0655e6fa604bedfd60e85c5bcb71c4cc23f6eb060d4d668393f0a" Nov 01 05:30:19.375483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.375466 4425 scope.go:122] "RemoveContainer" containerID="22176c352ce79d2601520409227322e0fe73f86b9bd140c2626178638643453a" Nov 01 05:30:19.384237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.384218 4425 scope.go:122] "RemoveContainer" containerID="ba8e88d132a402bb0b1fc5ac27f2ddfe27475f66c820841f00b058463cce8eed" Nov 01 05:30:19.458900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.458871 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:19.577207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.577171 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:19.618949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.618918 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:19.750974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.750917 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:19.752824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.752797 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\" (UniqueName: \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\") pod \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " Nov 01 05:30:19.752965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.752930 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\") pod \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\" (UID: \"f11915e9-cafa-4c38-bfd6-c08b2d672fc9\") " Nov 01 05:30:19.753307 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.753283 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca" pod "f11915e9-cafa-4c38-bfd6-c08b2d672fc9" (UID: "f11915e9-cafa-4c38-bfd6-c08b2d672fc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:19.753452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.753437 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.755361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.755338 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf" pod "f11915e9-cafa-4c38-bfd6-c08b2d672fc9" (UID: "f11915e9-cafa-4c38-bfd6-c08b2d672fc9"). InnerVolumeSpecName "kube-api-access-99ztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:19.755658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.755638 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891850-gqzk8"] Nov 01 05:30:19.854404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.854375 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99ztf\" (UniqueName: \"kubernetes.io/projected/f11915e9-cafa-4c38-bfd6-c08b2d672fc9-kube-api-access-99ztf\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:19.869874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.869837 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:19.872876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.872852 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891850-t57b8"] Nov 01 05:30:19.991923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.991887 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:19.996384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:19.996355 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-918b8d-29891850-mn2w2"] Nov 01 05:30:20.118797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.118717 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:20.121576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.121554 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891850-lgmxn"] Nov 01 05:30:20.362489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.362462 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" Nov 01 05:30:20.362925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.362462 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s" event={"ID":"f11915e9-cafa-4c38-bfd6-c08b2d672fc9","Type":"ContainerDied","Data":"b6fd69ce0bef3dfb81260eca7459664e1cdf6e81b0277993509f317729335b63"} Nov 01 05:30:20.362925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.362588 4425 scope.go:122] "RemoveContainer" containerID="ee8e6c2f86b4b823bf86e64654bbbb96a642e69253dc386059cb84126c8bc7f3" Nov 01 05:30:20.365558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.365534 4425 generic.go:356] "Generic (PLEG): container finished" podID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerID="880a81498833823e9b2c5836c2057619fa356f426177fd123ab79c71dfb8dea3" exitCode=60 Nov 01 05:30:20.365709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.365617 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerDied","Data":"880a81498833823e9b2c5836c2057619fa356f426177fd123ab79c71dfb8dea3"} Nov 01 05:30:20.366073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.366050 4425 scope.go:122] "RemoveContainer" containerID="880a81498833823e9b2c5836c2057619fa356f426177fd123ab79c71dfb8dea3" Nov 01 05:30:20.366436 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:20.366398 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5_odh-ai-gateway-infra(c1c981ce-7b33-4ec9-a9c0-fff070e31929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" Nov 01 05:30:20.373480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.373463 4425 scope.go:122] "RemoveContainer" containerID="ed3ea12ad42d7de25b44ae4fa2d433e703beb8d131f6b1d4d0b617e77a3073c3" Nov 01 05:30:20.526970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.526944 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:20.527982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.527964 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:20.534484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.534460 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:20.541074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.541052 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891850-p6q9s"] Nov 01 05:30:20.665662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.665582 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\" (UniqueName: \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\") pod \"cd47d121-6d91-4e9d-b497-586dc931ce80\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " Nov 01 05:30:20.665829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.665809 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\") pod \"cd47d121-6d91-4e9d-b497-586dc931ce80\" (UID: \"cd47d121-6d91-4e9d-b497-586dc931ce80\") " Nov 01 05:30:20.666154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.666127 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca" pod "cd47d121-6d91-4e9d-b497-586dc931ce80" (UID: "cd47d121-6d91-4e9d-b497-586dc931ce80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:20.666354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.666338 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd47d121-6d91-4e9d-b497-586dc931ce80-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:20.668223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.668198 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7" pod "cd47d121-6d91-4e9d-b497-586dc931ce80" (UID: "cd47d121-6d91-4e9d-b497-586dc931ce80"). InnerVolumeSpecName "kube-api-access-mvmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:20.767912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.767878 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmd7\" (UniqueName: \"kubernetes.io/projected/cd47d121-6d91-4e9d-b497-586dc931ce80-kube-api-access-mvmd7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:20.977702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.977610 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" path="/var/lib/kubelet/pods/69493047-8928-41c8-ae8c-12238aad3dd9/volumes" Nov 01 05:30:20.978057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.978042 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" path="/var/lib/kubelet/pods/9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7/volumes" Nov 01 05:30:20.978462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.978448 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" path="/var/lib/kubelet/pods/ad72f17a-1ccf-47c8-b80b-6f733afaf1fe/volumes" Nov 01 05:30:20.979121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.979106 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" path="/var/lib/kubelet/pods/b13c484c-ea80-434e-b773-dcb061d70eb1/volumes" Nov 01 05:30:20.979534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:20.979519 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" path="/var/lib/kubelet/pods/f11915e9-cafa-4c38-bfd6-c08b2d672fc9/volumes" Nov 01 05:30:21.379907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.379875 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" event={"ID":"cd47d121-6d91-4e9d-b497-586dc931ce80","Type":"ContainerDied","Data":"de2d2c5947cb893f02f7730cb178ee25c294fe5c61bfea8cf110ac416f5860b5"} Nov 01 05:30:21.380274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.379920 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w" Nov 01 05:30:21.380274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.379938 4425 scope.go:122] "RemoveContainer" containerID="953570386f9ad245a14404136866458d29e7a51c62033f863756e8054b174fb6" Nov 01 05:30:21.541955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.541919 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:21.648763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.648682 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:21.664394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.664357 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891850-bj46w"] Nov 01 05:30:21.678733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.678703 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\") pod \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " Nov 01 05:30:21.678882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.678801 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\") pod \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\" (UID: \"c1c981ce-7b33-4ec9-a9c0-fff070e31929\") " Nov 01 05:30:21.679140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.679113 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca" pod "c1c981ce-7b33-4ec9-a9c0-fff070e31929" (UID: "c1c981ce-7b33-4ec9-a9c0-fff070e31929"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:21.679320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.679303 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c981ce-7b33-4ec9-a9c0-fff070e31929-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:21.681091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.681069 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4" pod "c1c981ce-7b33-4ec9-a9c0-fff070e31929" (UID: "c1c981ce-7b33-4ec9-a9c0-fff070e31929"). InnerVolumeSpecName "kube-api-access-sf6t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:21.781187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:21.781147 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6t4\" (UniqueName: \"kubernetes.io/projected/c1c981ce-7b33-4ec9-a9c0-fff070e31929-kube-api-access-sf6t4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:22.387919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.387890 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" Nov 01 05:30:22.387919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.387911 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5" event={"ID":"c1c981ce-7b33-4ec9-a9c0-fff070e31929","Type":"ContainerDied","Data":"62b9e9f719c2dbe38bbb34f17c31f3864dc4fe1ed1793c45b35c55710b81f38a"} Nov 01 05:30:22.388491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.387988 4425 scope.go:122] "RemoveContainer" containerID="880a81498833823e9b2c5836c2057619fa356f426177fd123ab79c71dfb8dea3" Nov 01 05:30:22.560619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.560589 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:22.611744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.611711 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891850-ndjn5"] Nov 01 05:30:22.973898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.973867 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" path="/var/lib/kubelet/pods/c1c981ce-7b33-4ec9-a9c0-fff070e31929/volumes" Nov 01 05:30:22.974306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:22.974292 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" path="/var/lib/kubelet/pods/cd47d121-6d91-4e9d-b497-586dc931ce80/volumes" Nov 01 05:30:26.948306 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:26.948242 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:30:27.302210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302134 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2"] Nov 01 05:30:27.302740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302725 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302742 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302753 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302758 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302771 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302776 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302782 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.302784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302786 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302792 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302796 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302810 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302815 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302822 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302827 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302833 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302836 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302843 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302847 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302856 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302860 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302867 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302871 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302876 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302880 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302889 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302893 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302899 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302906 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302916 4425 cpu_manager.go:414] "containerMap: removing container" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302920 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302925 4425 cpu_manager.go:414] "containerMap: removing container" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302929 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302940 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302944 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302950 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302954 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302963 4425 cpu_manager.go:414] "containerMap: removing container" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.302995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.302966 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303053 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303059 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303064 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303070 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303076 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303080 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303085 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="69493047-8928-41c8-ae8c-12238aad3dd9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303091 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303096 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303102 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303106 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303110 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303115 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303120 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303124 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303130 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303136 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11915e9-cafa-4c38-bfd6-c08b2d672fc9" containerName="cleanup" Nov 01 05:30:27.303878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.303141 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:27.306349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.306333 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.347071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.347034 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.347224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.347150 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcx42\" (UniqueName: \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.347224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.347210 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.364015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.363981 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2"] Nov 01 05:30:27.374724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.374701 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:30:27.411061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.411026 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:30:27.411061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.411033 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:30:27.411236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.411109 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:30:27.448698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.448657 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcx42\" (UniqueName: \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.448853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.448720 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.448917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.448857 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.449547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.449487 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.451294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.451271 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.511938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.511909 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcx42\" (UniqueName: \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\") pod \"maas-api-revoke-sub-076880c2-nfjl2\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.619327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.619294 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:27.998033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:27.998007 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2"] Nov 01 05:30:27.999691 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:27.999662 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a70d4a_6741_4247_b09a_a5e49c21a653.slice/crio-d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9 WatchSource:0}: Error finding container d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9: Status 404 returned error can't find the container with id d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9 Nov 01 05:30:28.162863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:28.162833 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:30:28.163074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:28.163049 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" containerID="cri-o://66612c6c6efcd6807a1d3cfb41e294988db17aa14b538e8711c639945b9ddbac" gracePeriod=30 Nov 01 05:30:28.435096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:28.435064 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" event={"ID":"40a70d4a-6741-4247-b09a-a5e49c21a653","Type":"ContainerStarted","Data":"71934044f8bb335199b1d4a4965e8654ad2ed6a22e86ad085c4f92a40a74577d"} Nov 01 05:30:28.435304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:28.435118 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" event={"ID":"40a70d4a-6741-4247-b09a-a5e49c21a653","Type":"ContainerStarted","Data":"d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9"} Nov 01 05:30:28.553973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:28.553917 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" podStartSLOduration=1.553902398 podStartE2EDuration="1.553902398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:28.55297644 +0000 UTC m=+1643.693565376" watchObservedRunningTime="2026-11-01 05:30:28.553902398 +0000 UTC m=+1643.694491350" Nov 01 05:30:29.452688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:29.452649 4425 generic.go:356] "Generic (PLEG): container finished" podID="40a70d4a-6741-4247-b09a-a5e49c21a653" containerID="71934044f8bb335199b1d4a4965e8654ad2ed6a22e86ad085c4f92a40a74577d" exitCode=0 Nov 01 05:30:29.453229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:29.452742 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" event={"ID":"40a70d4a-6741-4247-b09a-a5e49c21a653","Type":"ContainerDied","Data":"71934044f8bb335199b1d4a4965e8654ad2ed6a22e86ad085c4f92a40a74577d"} Nov 01 05:30:30.630031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.630003 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:30.691826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.691781 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\") pod \"40a70d4a-6741-4247-b09a-a5e49c21a653\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " Nov 01 05:30:30.692037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.691851 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\" (UniqueName: \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\") pod \"40a70d4a-6741-4247-b09a-a5e49c21a653\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " Nov 01 05:30:30.692037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.691928 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\") pod \"40a70d4a-6741-4247-b09a-a5e49c21a653\" (UID: \"40a70d4a-6741-4247-b09a-a5e49c21a653\") " Nov 01 05:30:30.692217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.692195 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca" pod "40a70d4a-6741-4247-b09a-a5e49c21a653" (UID: "40a70d4a-6741-4247-b09a-a5e49c21a653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:30.694220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.694192 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42" pod "40a70d4a-6741-4247-b09a-a5e49c21a653" (UID: "40a70d4a-6741-4247-b09a-a5e49c21a653"). InnerVolumeSpecName "kube-api-access-dcx42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:30.694346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.694216 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca" pod "40a70d4a-6741-4247-b09a-a5e49c21a653" (UID: "40a70d4a-6741-4247-b09a-a5e49c21a653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:30:30.793995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.793891 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:30.793995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.793928 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcx42\" (UniqueName: \"kubernetes.io/projected/40a70d4a-6741-4247-b09a-a5e49c21a653-kube-api-access-dcx42\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:30.793995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:30.793938 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a70d4a-6741-4247-b09a-a5e49c21a653-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:31.470327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:31.470286 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" event={"ID":"40a70d4a-6741-4247-b09a-a5e49c21a653","Type":"ContainerDied","Data":"d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9"} Nov 01 05:30:31.470529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:31.470349 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e852f5abca6680cc238558d5f4ccbca081c49c226130bbc4a93b45224582e9" Nov 01 05:30:31.470529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:31.470295 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2" Nov 01 05:30:32.913326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:32.913290 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:30:32.913755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:32.913348 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:30:33.505579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.505474 4425 generic.go:356] "Generic (PLEG): container finished" podID="a949816b-9a7e-4580-a380-73f1262abda3" containerID="66612c6c6efcd6807a1d3cfb41e294988db17aa14b538e8711c639945b9ddbac" exitCode=0 Nov 01 05:30:33.505753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.505576 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" event={"ID":"a949816b-9a7e-4580-a380-73f1262abda3","Type":"ContainerDied","Data":"66612c6c6efcd6807a1d3cfb41e294988db17aa14b538e8711c639945b9ddbac"} Nov 01 05:30:33.694309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.694280 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:30:33.747738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747706 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.747738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747745 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.747989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747807 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.747989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747851 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.747989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747915 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.747989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.747943 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.748205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748036 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.748205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748099 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:30:33.748341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748200 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.748396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748344 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\") pod \"a949816b-9a7e-4580-a380-73f1262abda3\" (UID: \"a949816b-9a7e-4580-a380-73f1262abda3\") " Nov 01 05:30:33.748396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748375 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:33.748487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748445 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:30:33.748731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748707 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:30:33.748796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748749 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:30:33.748796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748776 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.748890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748795 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.748890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.748809 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949816b-9a7e-4580-a380-73f1262abda3-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.751015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.750894 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "kube-api-access-8b6j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:33.751015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.750972 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:30:33.751015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.750982 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:30:33.751189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.751045 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token" pod "a949816b-9a7e-4580-a380-73f1262abda3" (UID: "a949816b-9a7e-4580-a380-73f1262abda3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:33.850625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850593 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.850625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850622 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.850625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850632 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6j6\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-kube-api-access-8b6j6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.850890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850642 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949816b-9a7e-4580-a380-73f1262abda3-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.850890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850651 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949816b-9a7e-4580-a380-73f1262abda3-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:33.850890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:33.850659 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949816b-9a7e-4580-a380-73f1262abda3-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:34.517695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.517653 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" event={"ID":"a949816b-9a7e-4580-a380-73f1262abda3","Type":"ContainerDied","Data":"98c7fdde10883a4a43f9e6faa3809f1e5d3d47a52803973160f0c78b3f5e07e7"} Nov 01 05:30:34.518170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.517706 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb" Nov 01 05:30:34.518170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.517734 4425 scope.go:122] "RemoveContainer" containerID="66612c6c6efcd6807a1d3cfb41e294988db17aa14b538e8711c639945b9ddbac" Nov 01 05:30:34.688194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.688165 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:30:34.691284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.691235 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-7gfxb"] Nov 01 05:30:34.973745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:34.973714 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a949816b-9a7e-4580-a380-73f1262abda3" path="/var/lib/kubelet/pods/a949816b-9a7e-4580-a380-73f1262abda3/volumes" Nov 01 05:30:37.455537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.455506 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:30:37.456077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456064 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" Nov 01 05:30:37.456131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456078 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" Nov 01 05:30:37.456131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456104 4425 cpu_manager.go:414] "containerMap: removing container" podUID="40a70d4a-6741-4247-b09a-a5e49c21a653" containerName="revoke-keys" Nov 01 05:30:37.456131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456109 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a70d4a-6741-4247-b09a-a5e49c21a653" containerName="revoke-keys" Nov 01 05:30:37.456131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456123 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:37.456131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456129 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7bdb3c-c8e6-40c8-9887-04325a5e3aa7" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456149 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456154 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad72f17a-1ccf-47c8-b80b-6f733afaf1fe" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456234 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd47d121-6d91-4e9d-b497-586dc931ce80" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456241 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c981ce-7b33-4ec9-a9c0-fff070e31929" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456263 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a949816b-9a7e-4580-a380-73f1262abda3" containerName="istio-proxy" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456270 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13c484c-ea80-434e-b773-dcb061d70eb1" containerName="cleanup" Nov 01 05:30:37.456343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.456282 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a70d4a-6741-4247-b09a-a5e49c21a653" containerName="revoke-keys" Nov 01 05:30:37.461148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.461114 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.519515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.519482 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:30:37.566384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566344 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-17306a2b8c16\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566438 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566438 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-584534d3ba60\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566438 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-49dcf695f990\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566469 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566458 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\"" Nov 01 05:30:37.566549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.566466 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-f8c698a71031\"" Nov 01 05:30:37.567948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.567932 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\"" Nov 01 05:30:37.567948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.567944 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\"" Nov 01 05:30:37.568039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.567970 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\"" Nov 01 05:30:37.598062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.598022 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.598062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.598061 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.598303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.598243 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.598369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.598351 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q66g\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.699887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.699850 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.699887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.699888 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.700205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.700000 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.700205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.700040 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q66g\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.700760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.700733 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.700877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.700842 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.702955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.702931 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.766198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.766123 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q66g\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:37.774965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:37.774938 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:38.152544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:38.152513 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:30:38.153418 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:38.153389 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f8b27a_a55e_466b_8895_e4b6aaf503f1.slice/crio-e8e99e6275114c4b2fe5a28d9f345664fea2e3933ddd20f3c80e87134aa10c7d WatchSource:0}: Error finding container e8e99e6275114c4b2fe5a28d9f345664fea2e3933ddd20f3c80e87134aa10c7d: Status 404 returned error can't find the container with id e8e99e6275114c4b2fe5a28d9f345664fea2e3933ddd20f3c80e87134aa10c7d Nov 01 05:30:38.547101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:38.547008 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" event={"ID":"44f8b27a-a55e-466b-8895-e4b6aaf503f1","Type":"ContainerStarted","Data":"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101"} Nov 01 05:30:38.547101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:38.547071 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" event={"ID":"44f8b27a-a55e-466b-8895-e4b6aaf503f1","Type":"ContainerStarted","Data":"e8e99e6275114c4b2fe5a28d9f345664fea2e3933ddd20f3c80e87134aa10c7d"} Nov 01 05:30:38.547548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:38.547203 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:38.667092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:38.667039 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" podStartSLOduration=1.6670260479999999 podStartE2EDuration="1.667026048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:38.666440273 +0000 UTC m=+1653.807029211" watchObservedRunningTime="2026-11-01 05:30:38.667026048 +0000 UTC m=+1653.807614966" Nov 01 05:30:41.354676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.354644 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc"] Nov 01 05:30:41.358980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.358959 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.417033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.416997 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc"] Nov 01 05:30:41.446936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.446890 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.447100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.447046 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.447185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.447105 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjnb\" (UniqueName: \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.463517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.463495 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:30:41.463779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.463764 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:30:41.463779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.463766 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:30:41.463918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.463830 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:30:41.548063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.548024 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.548246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.548076 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjnb\" (UniqueName: \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.548246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.548184 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.548855 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.548828 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.550690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.550668 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.611365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.611247 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjnb\" (UniqueName: \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\") pod \"maas-api-revoke-sub-eec49167-pzjcc\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:41.673241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:41.673206 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:42.054334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:42.054304 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc"] Nov 01 05:30:42.055638 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:42.055610 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8beec8_a107_4835_be2f_a96cbcb30ed7.slice/crio-1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23 WatchSource:0}: Error finding container 1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23: Status 404 returned error can't find the container with id 1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23 Nov 01 05:30:42.582880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:42.582847 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" event={"ID":"ae8beec8-a107-4835-be2f-a96cbcb30ed7","Type":"ContainerStarted","Data":"93fa515608fe45f66e48bf815f2edb64c0287690ec6780ed67ec0dc0cd1e35d0"} Nov 01 05:30:42.583296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:42.582923 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" event={"ID":"ae8beec8-a107-4835-be2f-a96cbcb30ed7","Type":"ContainerStarted","Data":"1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23"} Nov 01 05:30:42.702631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:42.702531 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" podStartSLOduration=1.702517751 podStartE2EDuration="1.702517751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:42.701916842 +0000 UTC m=+1657.842505779" watchObservedRunningTime="2026-11-01 05:30:42.702517751 +0000 UTC m=+1657.843106670" Nov 01 05:30:43.593363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:43.593322 4425 generic.go:356] "Generic (PLEG): container finished" podID="ae8beec8-a107-4835-be2f-a96cbcb30ed7" containerID="93fa515608fe45f66e48bf815f2edb64c0287690ec6780ed67ec0dc0cd1e35d0" exitCode=0 Nov 01 05:30:43.593797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:43.593405 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" event={"ID":"ae8beec8-a107-4835-be2f-a96cbcb30ed7","Type":"ContainerDied","Data":"93fa515608fe45f66e48bf815f2edb64c0287690ec6780ed67ec0dc0cd1e35d0"} Nov 01 05:30:44.765418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.765382 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:44.891844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.891765 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\" (UniqueName: \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\") pod \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " Nov 01 05:30:44.892021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.891905 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\") pod \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " Nov 01 05:30:44.892021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.891963 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\") pod \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\" (UID: \"ae8beec8-a107-4835-be2f-a96cbcb30ed7\") " Nov 01 05:30:44.892366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.892340 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca" pod "ae8beec8-a107-4835-be2f-a96cbcb30ed7" (UID: "ae8beec8-a107-4835-be2f-a96cbcb30ed7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:44.892528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.892502 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:44.894554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.894526 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb" pod "ae8beec8-a107-4835-be2f-a96cbcb30ed7" (UID: "ae8beec8-a107-4835-be2f-a96cbcb30ed7"). InnerVolumeSpecName "kube-api-access-6qjnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:44.894675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.894587 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca" pod "ae8beec8-a107-4835-be2f-a96cbcb30ed7" (UID: "ae8beec8-a107-4835-be2f-a96cbcb30ed7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:30:44.993927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.993895 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8beec8-a107-4835-be2f-a96cbcb30ed7-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:44.993927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:44.993924 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjnb\" (UniqueName: \"kubernetes.io/projected/ae8beec8-a107-4835-be2f-a96cbcb30ed7-kube-api-access-6qjnb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:45.609405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:45.609371 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" Nov 01 05:30:45.609405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:45.609392 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc" event={"ID":"ae8beec8-a107-4835-be2f-a96cbcb30ed7","Type":"ContainerDied","Data":"1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23"} Nov 01 05:30:45.609649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:45.609458 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1593f5094a134db847f0134bd696fc9b95b419bcf37ed227fb83d90c78582e23" Nov 01 05:30:49.556840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:49.556769 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:30:51.906426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.906396 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:30:51.907013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.906997 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ae8beec8-a107-4835-be2f-a96cbcb30ed7" containerName="revoke-keys" Nov 01 05:30:51.907061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.907014 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8beec8-a107-4835-be2f-a96cbcb30ed7" containerName="revoke-keys" Nov 01 05:30:51.907112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.907102 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8beec8-a107-4835-be2f-a96cbcb30ed7" containerName="revoke-keys" Nov 01 05:30:51.910633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.910618 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978060 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978126 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978241 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978296 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978327 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxv2\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978355 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978498 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978527 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:51.978704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:51.978583 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.012931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.012896 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:30:52.079726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.079677 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.079921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.079734 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxv2\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.079921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.079770 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.079921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.079849 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.079921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.079874 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080045 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080118 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080163 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080235 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080315 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080445 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080559 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080663 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.080937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.080907 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.082481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.082459 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.082817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.082795 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.096854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.096821 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn"] Nov 01 05:30:52.102401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.102374 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.142435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.142386 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.142601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.142572 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxv2\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.157475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.157404 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:30:52.157713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.157681 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:30:52.158078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.157751 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:30:52.158078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.157500 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:30:52.159004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.158975 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn"] Nov 01 05:30:52.181152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.181103 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.181385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.181361 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgq6\" (UniqueName: \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.181453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.181398 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.224088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.224048 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:52.282970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.282935 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.283127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.283093 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgq6\" (UniqueName: \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.283127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.283125 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.284016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.283992 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.300009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.299975 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.345725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.345694 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgq6\" (UniqueName: \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\") pod \"maas-api-revoke-sub-3bb58d33-zccmn\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.416969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.416887 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:52.656052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.656022 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:30:52.657101 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:52.657071 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9785f724_7bbc_4a8f_b065_1b3a23ac8a79.slice/crio-50b6a7d4bacc5588f4a1f4069742993ab65836d88e5d767b1f0f1fe66335514e WatchSource:0}: Error finding container 50b6a7d4bacc5588f4a1f4069742993ab65836d88e5d767b1f0f1fe66335514e: Status 404 returned error can't find the container with id 50b6a7d4bacc5588f4a1f4069742993ab65836d88e5d767b1f0f1fe66335514e Nov 01 05:30:52.659685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.659647 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:30:52.659798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.659737 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:30:52.659798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.659781 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:30:52.796583 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:30:52.796554 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334fd691_5f20_411d_9de9_45a504a8f5c4.slice/crio-c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300 WatchSource:0}: Error finding container c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300: Status 404 returned error can't find the container with id c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300 Nov 01 05:30:52.797887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:52.797865 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn"] Nov 01 05:30:53.675571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.675535 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" event={"ID":"9785f724-7bbc-4a8f-b065-1b3a23ac8a79","Type":"ContainerStarted","Data":"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7"} Nov 01 05:30:53.676016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.675589 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" event={"ID":"9785f724-7bbc-4a8f-b065-1b3a23ac8a79","Type":"ContainerStarted","Data":"50b6a7d4bacc5588f4a1f4069742993ab65836d88e5d767b1f0f1fe66335514e"} Nov 01 05:30:53.677365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.677335 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" event={"ID":"334fd691-5f20-411d-9de9-45a504a8f5c4","Type":"ContainerStarted","Data":"4e7149686957d5eaaff681ac3f0c877599a67a9606625659622f59662c286516"} Nov 01 05:30:53.677443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.677377 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" event={"ID":"334fd691-5f20-411d-9de9-45a504a8f5c4","Type":"ContainerStarted","Data":"c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300"} Nov 01 05:30:53.869016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.868945 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podStartSLOduration=2.868921579 podStartE2EDuration="2.868921579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:53.862677061 +0000 UTC m=+1669.003266019" watchObservedRunningTime="2026-11-01 05:30:53.868921579 +0000 UTC m=+1669.009510499" Nov 01 05:30:53.981804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:53.981676 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" podStartSLOduration=1.981661494 podStartE2EDuration="1.981661494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:53.98080197 +0000 UTC m=+1669.121390890" watchObservedRunningTime="2026-11-01 05:30:53.981661494 +0000 UTC m=+1669.122250413" Nov 01 05:30:54.224599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:54.224557 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:30:54.226125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:54.226100 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:54.226235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:54.226158 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:30:54.687083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:54.687045 4425 generic.go:356] "Generic (PLEG): container finished" podID="334fd691-5f20-411d-9de9-45a504a8f5c4" containerID="4e7149686957d5eaaff681ac3f0c877599a67a9606625659622f59662c286516" exitCode=0 Nov 01 05:30:54.687504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:54.687128 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" event={"ID":"334fd691-5f20-411d-9de9-45a504a8f5c4","Type":"ContainerDied","Data":"4e7149686957d5eaaff681ac3f0c877599a67a9606625659622f59662c286516"} Nov 01 05:30:55.224533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:55.224493 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:55.224751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:55.224547 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:30:55.906196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:55.906132 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:56.032210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.032111 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\") pod \"334fd691-5f20-411d-9de9-45a504a8f5c4\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " Nov 01 05:30:56.032425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.032243 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\") pod \"334fd691-5f20-411d-9de9-45a504a8f5c4\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " Nov 01 05:30:56.032425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.032385 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\" (UniqueName: \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\") pod \"334fd691-5f20-411d-9de9-45a504a8f5c4\" (UID: \"334fd691-5f20-411d-9de9-45a504a8f5c4\") " Nov 01 05:30:56.032639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.032600 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca" pod "334fd691-5f20-411d-9de9-45a504a8f5c4" (UID: "334fd691-5f20-411d-9de9-45a504a8f5c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:30:56.032934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.032916 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:56.034920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.034897 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca" pod "334fd691-5f20-411d-9de9-45a504a8f5c4" (UID: "334fd691-5f20-411d-9de9-45a504a8f5c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:30:56.035070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.035054 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6" pod "334fd691-5f20-411d-9de9-45a504a8f5c4" (UID: "334fd691-5f20-411d-9de9-45a504a8f5c4"). InnerVolumeSpecName "kube-api-access-vfgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:30:56.133908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.133871 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgq6\" (UniqueName: \"kubernetes.io/projected/334fd691-5f20-411d-9de9-45a504a8f5c4-kube-api-access-vfgq6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:56.133908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.133903 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334fd691-5f20-411d-9de9-45a504a8f5c4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:30:56.224971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.224933 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:56.225181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.225005 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:30:56.705315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.705278 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" event={"ID":"334fd691-5f20-411d-9de9-45a504a8f5c4","Type":"ContainerDied","Data":"c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300"} Nov 01 05:30:56.705315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.705306 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn" Nov 01 05:30:56.705531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:56.705331 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c151ce832e02422206bb2321135f59d463f280aca915b8e0fab7628df3d03300" Nov 01 05:30:56.972288 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:30:56.972181 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:30:57.225369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:57.225240 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:57.225369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:57.225341 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:30:58.225141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:58.225095 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:58.225579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:58.225176 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:30:59.224372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:59.224341 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 05:30:59.224553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:30:59.224406 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 05:31:00.230214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:00.230186 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:31:00.230663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:00.230575 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:31:00.231356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:00.231338 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:31:03.679082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.679003 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:31:03.680488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.680443 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" containerID="cri-o://5ee2d7e73c34c25c8fbaf3690252006fe78daebf5e2570114e5df23c806f4146" gracePeriod=30 Nov 01 05:31:03.732407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.732352 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:31:03.732712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.732676 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" containerID="cri-o://f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334" gracePeriod=30 Nov 01 05:31:03.732894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.732466 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:31:03.733226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.733089 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" containerID="cri-o://9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c" gracePeriod=30 Nov 01 05:31:03.789280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.783966 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:31:03.789280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:03.784313 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" containerID="cri-o://fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f" gracePeriod=30 Nov 01 05:31:06.300806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.300772 4425 scope.go:122] "RemoveContainer" containerID="63f1366581f52b85f0d76ce8fa8f12452c8d901bd587fe93b2a9445ad0ca2e7e" Nov 01 05:31:06.312224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.312199 4425 scope.go:122] "RemoveContainer" containerID="44bd89f4ff0712d6ee3396edf0f2cbf619979302a1ba7790474ba0478af5b04a" Nov 01 05:31:06.324021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.323880 4425 scope.go:122] "RemoveContainer" containerID="1dc8fcf78b713bd97b7f6f795799f5c3d180e3d5121e6335333951389d41e91b" Nov 01 05:31:06.337139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.337104 4425 scope.go:122] "RemoveContainer" containerID="f550518ad49021ed599b257f453c3f9fce9a480275a1385f76958723e7aea0ae" Nov 01 05:31:06.554189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.554106 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-94m8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:31:06.554356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:06.554190 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:31:08.200077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:08.200038 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-7g89g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:31:08.200760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:08.200111 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:31:08.813696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:08.813645 4425 generic.go:356] "Generic (PLEG): container finished" podID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerID="5ee2d7e73c34c25c8fbaf3690252006fe78daebf5e2570114e5df23c806f4146" exitCode=0 Nov 01 05:31:08.813905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:08.813881 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" event={"ID":"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0","Type":"ContainerDied","Data":"5ee2d7e73c34c25c8fbaf3690252006fe78daebf5e2570114e5df23c806f4146"} Nov 01 05:31:09.104346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.104317 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:31:09.150612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.150539 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:31:09.156702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.156651 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:31:09.210479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.210452 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:31:09.261406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261323 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261396 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261432 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261472 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261496 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261526 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261574 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261602 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261650 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261699 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261752 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261750 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261784 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261821 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261848 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.261932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261924 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.262544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261956 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.261987 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.262544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262003 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.262544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262021 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262565 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:09.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262618 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262704 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262745 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\") pod \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\" (UID: \"d26dc12f-9951-4c09-8fa7-52e84ab40c20\") " Nov 01 05:31:09.262990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262818 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262848 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262880 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262912 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.262990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.262967 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\") pod \"718726c9-f7d8-4cd2-b064-0d327136ff1e\" (UID: \"718726c9-f7d8-4cd2-b064-0d327136ff1e\") " Nov 01 05:31:09.263293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263011 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\") pod \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\" (UID: \"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0\") " Nov 01 05:31:09.263973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263678 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:09.263973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263876 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.263973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263953 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.263973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263971 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.264282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.263987 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.264282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264001 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.264282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264225 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.264474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264446 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264539 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264684 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264700 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264795 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264884 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264921 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.264990 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.265536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.265292 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.266102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.266070 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:09.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.266153 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:09.266329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.266307 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.266480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.266456 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "kube-api-access-v98t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.267204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.267086 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.267204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.267179 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.267610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.267586 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "kube-api-access-pkl5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.267699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.267589 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "kube-api-access-864rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.267699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.267623 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.268179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.268161 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy" pod "718726c9-f7d8-4cd2-b064-0d327136ff1e" (UID: "718726c9-f7d8-4cd2-b064-0d327136ff1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.268318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.268301 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token" pod "d26dc12f-9951-4c09-8fa7-52e84ab40c20" (UID: "d26dc12f-9951-4c09-8fa7-52e84ab40c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.268534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.268521 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo" pod "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" (UID: "fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:09.364911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.364867 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.364950 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365023 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365167 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365213 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.365286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365245 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365269 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.365487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365334 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365378 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365437 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365464 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.365487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365474 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\") pod \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\" (UID: \"90a256d4-691c-4148-bf6a-f9cf0d04e8fe\") " Nov 01 05:31:09.365773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365593 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:09.365773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.365720 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.366306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366246 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v98t6\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-kube-api-access-v98t6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366294 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366310 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366324 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366338 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366353 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366365 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366380 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366394 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkl5c\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-kube-api-access-pkl5c\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366407 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366422 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366435 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366448 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366462 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366477 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.366485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366490 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366507 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366524 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366541 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366553 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366568 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26dc12f-9951-4c09-8fa7-52e84ab40c20-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366582 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366596 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366611 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/718726c9-f7d8-4cd2-b064-0d327136ff1e-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366625 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864rm\" (UniqueName: \"kubernetes.io/projected/718726c9-f7d8-4cd2-b064-0d327136ff1e-kube-api-access-864rm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366641 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/718726c9-f7d8-4cd2-b064-0d327136ff1e-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366655 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.366667 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.367990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.367872 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "kube-api-access-4rqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.367990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.367938 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:09.368174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.368157 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:09.368231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.368182 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token" pod "90a256d4-691c-4148-bf6a-f9cf0d04e8fe" (UID: "90a256d4-691c-4148-bf6a-f9cf0d04e8fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:09.467342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.467300 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.467342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.467328 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqwp\" (UniqueName: \"kubernetes.io/projected/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-kube-api-access-4rqwp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.467342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.467339 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.467342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.467348 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a256d4-691c-4148-bf6a-f9cf0d04e8fe-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:09.821717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.821684 4425 generic.go:356] "Generic (PLEG): container finished" podID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerID="fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f" exitCode=0 Nov 01 05:31:09.821903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.821761 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" Nov 01 05:31:09.821903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.821774 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" event={"ID":"90a256d4-691c-4148-bf6a-f9cf0d04e8fe","Type":"ContainerDied","Data":"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f"} Nov 01 05:31:09.821903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.821835 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx" event={"ID":"90a256d4-691c-4148-bf6a-f9cf0d04e8fe","Type":"ContainerDied","Data":"73c830b3fff78a4fe2b33193de53a4823b593a1361b4db3f2854c55dd469761a"} Nov 01 05:31:09.821903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.821863 4425 scope.go:122] "RemoveContainer" containerID="fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f" Nov 01 05:31:09.823477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.823441 4425 generic.go:356] "Generic (PLEG): container finished" podID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerID="f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334" exitCode=0 Nov 01 05:31:09.823682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.823663 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" event={"ID":"718726c9-f7d8-4cd2-b064-0d327136ff1e","Type":"ContainerDied","Data":"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334"} Nov 01 05:31:09.823764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.823715 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" event={"ID":"718726c9-f7d8-4cd2-b064-0d327136ff1e","Type":"ContainerDied","Data":"2c1a61a995a3e3ed7378b064d32f3e5f6e340da96576fa2a74faaf1373bb7ef9"} Nov 01 05:31:09.823764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.823671 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv" Nov 01 05:31:09.825245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.825225 4425 generic.go:356] "Generic (PLEG): container finished" podID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerID="9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c" exitCode=0 Nov 01 05:31:09.825384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.825338 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" event={"ID":"d26dc12f-9951-4c09-8fa7-52e84ab40c20","Type":"ContainerDied","Data":"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c"} Nov 01 05:31:09.825474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.825381 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" event={"ID":"d26dc12f-9951-4c09-8fa7-52e84ab40c20","Type":"ContainerDied","Data":"ad3bc9144b364001262bfc7067400d4a6baa4d67bf96c4703d468a05d5bf366a"} Nov 01 05:31:09.825474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.825448 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g" Nov 01 05:31:09.831881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.829516 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" event={"ID":"fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0","Type":"ContainerDied","Data":"50d93cafcf30bdb3b7424cffa27ddb1bdd168bd70d31ab4d6bfea6d3c51d7742"} Nov 01 05:31:09.831881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.829618 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z" Nov 01 05:31:09.836432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.836408 4425 scope.go:122] "RemoveContainer" containerID="fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f" Nov 01 05:31:09.836851 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:09.836816 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f\": container with ID starting with fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f not found: ID does not exist" containerID="fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f" Nov 01 05:31:09.836931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.836865 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f"} err="failed to get container status \"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f\": rpc error: code = NotFound desc = could not find container \"fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f\": container with ID starting with fa0eac851b5f3ab096e84196eb580d13f3703c8c60795c52fca578f9285e268f not found: ID does not exist" Nov 01 05:31:09.836931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.836893 4425 scope.go:122] "RemoveContainer" containerID="f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334" Nov 01 05:31:09.851150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.851041 4425 scope.go:122] "RemoveContainer" containerID="f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334" Nov 01 05:31:09.851476 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:09.851453 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334\": container with ID starting with f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334 not found: ID does not exist" containerID="f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334" Nov 01 05:31:09.851561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.851487 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334"} err="failed to get container status \"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334\": rpc error: code = NotFound desc = could not find container \"f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334\": container with ID starting with f3cc0829d18b37bbdfc55fa6ae8d29bcdda67c2ec63e00232f33756b72ed5334 not found: ID does not exist" Nov 01 05:31:09.851561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.851507 4425 scope.go:122] "RemoveContainer" containerID="9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c" Nov 01 05:31:09.861156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.861131 4425 scope.go:122] "RemoveContainer" containerID="9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c" Nov 01 05:31:09.861485 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:09.861464 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c\": container with ID starting with 9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c not found: ID does not exist" containerID="9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c" Nov 01 05:31:09.861549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.861494 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c"} err="failed to get container status \"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c\": rpc error: code = NotFound desc = could not find container \"9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c\": container with ID starting with 9276eeffbe833f374800e0fbb28ea7714e5218a0c74a45caad4c21977e64078c not found: ID does not exist" Nov 01 05:31:09.861549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.861515 4425 scope.go:122] "RemoveContainer" containerID="5ee2d7e73c34c25c8fbaf3690252006fe78daebf5e2570114e5df23c806f4146" Nov 01 05:31:09.999665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:09.999587 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:31:10.005460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.005429 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tn2cx"] Nov 01 05:31:10.117407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.117375 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:31:10.123762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.123734 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-7g89g"] Nov 01 05:31:10.238324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.238287 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:31:10.243045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.243012 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-r5dmv"] Nov 01 05:31:10.352829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.352792 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:31:10.357481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.357452 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-94m8z"] Nov 01 05:31:10.665759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.665669 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4"] Nov 01 05:31:10.666529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666510 4425 cpu_manager.go:414] "containerMap: removing container" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666530 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666558 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666565 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666578 4425 cpu_manager.go:414] "containerMap: removing container" podUID="334fd691-5f20-411d-9de9-45a504a8f5c4" containerName="revoke-keys" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666587 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334fd691-5f20-411d-9de9-45a504a8f5c4" containerName="revoke-keys" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666618 4425 cpu_manager.go:414] "containerMap: removing container" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" Nov 01 05:31:10.666624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666625 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666635 4425 cpu_manager.go:414] "containerMap: removing container" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666642 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666771 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666781 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666799 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666810 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" containerName="istio-proxy" Nov 01 05:31:10.667016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.666820 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="334fd691-5f20-411d-9de9-45a504a8f5c4" containerName="revoke-keys" Nov 01 05:31:10.672314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.672287 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.726157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.726102 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4"] Nov 01 05:31:10.726394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.726281 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:31:10.726394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.726284 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:31:10.726394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.726376 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:31:10.726634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.726615 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:31:10.783106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.783067 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjkd\" (UniqueName: \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.783340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.783279 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.783422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.783402 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.884786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.884741 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjkd\" (UniqueName: \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.885004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.884863 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.885004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.884947 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.885559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.885530 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.887445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.887424 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.948018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.947940 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjkd\" (UniqueName: \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\") pod \"maas-api-revoke-sub-6570b548-2kfn4\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:10.973211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.973177 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718726c9-f7d8-4cd2-b064-0d327136ff1e" path="/var/lib/kubelet/pods/718726c9-f7d8-4cd2-b064-0d327136ff1e/volumes" Nov 01 05:31:10.973699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.973685 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a256d4-691c-4148-bf6a-f9cf0d04e8fe" path="/var/lib/kubelet/pods/90a256d4-691c-4148-bf6a-f9cf0d04e8fe/volumes" Nov 01 05:31:10.974086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.974072 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26dc12f-9951-4c09-8fa7-52e84ab40c20" path="/var/lib/kubelet/pods/d26dc12f-9951-4c09-8fa7-52e84ab40c20/volumes" Nov 01 05:31:10.974843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.974828 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0" path="/var/lib/kubelet/pods/fba2e5a0-c74e-45c6-a8aa-6dc5a3ca04c0/volumes" Nov 01 05:31:10.986141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:10.986108 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:11.366611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:11.366579 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4"] Nov 01 05:31:11.367355 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:31:11.367329 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf3345e_6458_46f3_b78b_9c4734f38fb2.slice/crio-ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b WatchSource:0}: Error finding container ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b: Status 404 returned error can't find the container with id ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b Nov 01 05:31:11.849224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:11.849192 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" event={"ID":"edf3345e-6458-46f3-b78b-9c4734f38fb2","Type":"ContainerStarted","Data":"c9c1b1ff7f0f650548827f4069c2cf13fb03fecdee17ce600f2c8000ed29f2c9"} Nov 01 05:31:11.849406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:11.849263 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" event={"ID":"edf3345e-6458-46f3-b78b-9c4734f38fb2","Type":"ContainerStarted","Data":"ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b"} Nov 01 05:31:11.968281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:11.968171 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" podStartSLOduration=1.9681579660000001 podStartE2EDuration="1.968157966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:11.967657073 +0000 UTC m=+1687.108245993" watchObservedRunningTime="2026-11-01 05:31:11.968157966 +0000 UTC m=+1687.108746884" Nov 01 05:31:12.856866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:12.856826 4425 generic.go:356] "Generic (PLEG): container finished" podID="edf3345e-6458-46f3-b78b-9c4734f38fb2" containerID="c9c1b1ff7f0f650548827f4069c2cf13fb03fecdee17ce600f2c8000ed29f2c9" exitCode=0 Nov 01 05:31:12.857335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:12.856906 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" event={"ID":"edf3345e-6458-46f3-b78b-9c4734f38fb2","Type":"ContainerDied","Data":"c9c1b1ff7f0f650548827f4069c2cf13fb03fecdee17ce600f2c8000ed29f2c9"} Nov 01 05:31:14.013755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.013731 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:14.048003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.047970 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r"] Nov 01 05:31:14.099201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.099170 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c87e06-d4s7r"] Nov 01 05:31:14.123237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.123150 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\") pod \"edf3345e-6458-46f3-b78b-9c4734f38fb2\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " Nov 01 05:31:14.123407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.123299 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\") pod \"edf3345e-6458-46f3-b78b-9c4734f38fb2\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " Nov 01 05:31:14.123407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.123375 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\" (UniqueName: \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\") pod \"edf3345e-6458-46f3-b78b-9c4734f38fb2\" (UID: \"edf3345e-6458-46f3-b78b-9c4734f38fb2\") " Nov 01 05:31:14.123672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.123640 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca" pod "edf3345e-6458-46f3-b78b-9c4734f38fb2" (UID: "edf3345e-6458-46f3-b78b-9c4734f38fb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:14.123785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.123772 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:14.125612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.125595 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca" pod "edf3345e-6458-46f3-b78b-9c4734f38fb2" (UID: "edf3345e-6458-46f3-b78b-9c4734f38fb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:31:14.125759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.125737 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd" pod "edf3345e-6458-46f3-b78b-9c4734f38fb2" (UID: "edf3345e-6458-46f3-b78b-9c4734f38fb2"). InnerVolumeSpecName "kube-api-access-gwjkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:14.224910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.224877 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf3345e-6458-46f3-b78b-9c4734f38fb2-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:14.224910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.224911 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjkd\" (UniqueName: \"kubernetes.io/projected/edf3345e-6458-46f3-b78b-9c4734f38fb2-kube-api-access-gwjkd\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:14.870314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.870279 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" Nov 01 05:31:14.870314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.870301 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4" event={"ID":"edf3345e-6458-46f3-b78b-9c4734f38fb2","Type":"ContainerDied","Data":"ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b"} Nov 01 05:31:14.870580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.870350 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe0c8e47201c991edd9eb9e77657123aded8baa64a7b1b05ab13120d0f8913b" Nov 01 05:31:14.974019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:14.973986 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfe56ff-f7c8-43cb-80f8-808f3f460da4" path="/var/lib/kubelet/pods/ccfe56ff-f7c8-43cb-80f8-808f3f460da4/volumes" Nov 01 05:31:16.190940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.190899 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq"] Nov 01 05:31:16.191506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.191490 4425 cpu_manager.go:414] "containerMap: removing container" podUID="edf3345e-6458-46f3-b78b-9c4734f38fb2" containerName="revoke-keys" Nov 01 05:31:16.191506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.191506 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf3345e-6458-46f3-b78b-9c4734f38fb2" containerName="revoke-keys" Nov 01 05:31:16.191645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.191600 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf3345e-6458-46f3-b78b-9c4734f38fb2" containerName="revoke-keys" Nov 01 05:31:16.195539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.195515 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.249335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.249305 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:31:16.249335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.249305 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:31:16.310915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.310883 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:31:16.311170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.310883 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:31:16.316470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.316391 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq"] Nov 01 05:31:16.350297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.350217 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.350559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.350319 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.350675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.350631 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkvn\" (UniqueName: \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.453988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.453893 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkvn\" (UniqueName: \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.454341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.454304 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.454461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.454358 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.455160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.455131 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.457584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.457563 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.516486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.516456 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkvn\" (UniqueName: \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\") pod \"maas-api-revoke-sub-006efc36-pv5wq\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:16.812543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:16.812515 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:17.194111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:17.194080 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq"] Nov 01 05:31:17.195277 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:31:17.195206 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77db025_c009_4cbe_9f41_64737b3b0ac6.slice/crio-6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d WatchSource:0}: Error finding container 6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d: Status 404 returned error can't find the container with id 6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d Nov 01 05:31:17.895876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:17.895834 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" event={"ID":"c77db025-c009-4cbe-9f41-64737b3b0ac6","Type":"ContainerStarted","Data":"c428a92f646d582161235ac697cd0258f4b282017380caef38d6868ad8633fa1"} Nov 01 05:31:17.896060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:17.895902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" event={"ID":"c77db025-c009-4cbe-9f41-64737b3b0ac6","Type":"ContainerStarted","Data":"6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d"} Nov 01 05:31:18.016787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:18.016720 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" podStartSLOduration=2.016698431 podStartE2EDuration="2.016698431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:18.014762946 +0000 UTC m=+1693.155351872" watchObservedRunningTime="2026-11-01 05:31:18.016698431 +0000 UTC m=+1693.157287350" Nov 01 05:31:18.498048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:18.498013 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:31:18.607764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:18.607727 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:31:18.907014 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:18.906982 4425 generic.go:356] "Generic (PLEG): container finished" podID="c77db025-c009-4cbe-9f41-64737b3b0ac6" containerID="c428a92f646d582161235ac697cd0258f4b282017380caef38d6868ad8633fa1" exitCode=0 Nov 01 05:31:18.907511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:18.907466 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" event={"ID":"c77db025-c009-4cbe-9f41-64737b3b0ac6","Type":"ContainerDied","Data":"c428a92f646d582161235ac697cd0258f4b282017380caef38d6868ad8633fa1"} Nov 01 05:31:20.090908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.090880 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:20.195077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.194991 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\" (UniqueName: \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\") pod \"c77db025-c009-4cbe-9f41-64737b3b0ac6\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " Nov 01 05:31:20.195077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.195062 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\") pod \"c77db025-c009-4cbe-9f41-64737b3b0ac6\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " Nov 01 05:31:20.195304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.195159 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\") pod \"c77db025-c009-4cbe-9f41-64737b3b0ac6\" (UID: \"c77db025-c009-4cbe-9f41-64737b3b0ac6\") " Nov 01 05:31:20.195518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.195490 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca" pod "c77db025-c009-4cbe-9f41-64737b3b0ac6" (UID: "c77db025-c009-4cbe-9f41-64737b3b0ac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:20.197496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.197472 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn" pod "c77db025-c009-4cbe-9f41-64737b3b0ac6" (UID: "c77db025-c009-4cbe-9f41-64737b3b0ac6"). InnerVolumeSpecName "kube-api-access-7tkvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:20.197496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.197474 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca" pod "c77db025-c009-4cbe-9f41-64737b3b0ac6" (UID: "c77db025-c009-4cbe-9f41-64737b3b0ac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:31:20.296729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.296693 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:20.296729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.296728 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tkvn\" (UniqueName: \"kubernetes.io/projected/c77db025-c009-4cbe-9f41-64737b3b0ac6-kube-api-access-7tkvn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:20.296941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.296744 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77db025-c009-4cbe-9f41-64737b3b0ac6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:20.924228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.924193 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" Nov 01 05:31:20.924228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.924218 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq" event={"ID":"c77db025-c009-4cbe-9f41-64737b3b0ac6","Type":"ContainerDied","Data":"6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d"} Nov 01 05:31:20.924458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:20.924304 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbf38918d1dd7c8bb7ef92ceb388380750d15ee869eab80b92cd8878c805f2d" Nov 01 05:31:21.906339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:21.906307 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:31:21.906902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:21.906886 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c77db025-c009-4cbe-9f41-64737b3b0ac6" containerName="revoke-keys" Nov 01 05:31:21.906963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:21.906904 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77db025-c009-4cbe-9f41-64737b3b0ac6" containerName="revoke-keys" Nov 01 05:31:21.907001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:21.906995 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77db025-c009-4cbe-9f41-64737b3b0ac6" containerName="revoke-keys" Nov 01 05:31:21.910467 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:21.910447 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015027 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015078 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015196 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015315 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015351 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015366 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015406 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dm2g\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015596 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.015688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.015646 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.018629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.018603 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:31:22.113933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.113892 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:31:22.117222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117196 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117305 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117372 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dm2g\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117465 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117489 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117498 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117544 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117574 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117624 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117721 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.117827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.117788 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.118054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.118034 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.118054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.118040 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.118506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.118034 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.118506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.118382 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.120327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.120303 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.120435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.120383 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172397 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-8eb61444db22\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172397 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172412 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172450 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172517 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172560 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-e2e--8ea5e171-dockercfg-hr8xq\"" Nov 01 05:31:22.172813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.172602 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\"" Nov 01 05:31:22.176288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.176245 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:31:22.183728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.183697 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.184004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.183982 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dm2g\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.218609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.218578 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.218730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.218637 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmrd\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.218766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.218726 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.218984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.218956 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.223284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.223263 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:22.319670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.319641 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.319870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.319730 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.319870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.319783 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmrd\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.319870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.319815 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.320434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.320401 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.320585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.320566 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.322587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.322566 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.382974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.382943 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmrd\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.453850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.453759 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.651168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.651141 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:31:22.651566 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:31:22.651545 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6270326e_b279_4e83_9876_ce4d9ab8af8f.slice/crio-7cbedf4e4e1a8c5f594d934ac6003c8cd51f487f3dbf610628804cc87c0d5c45 WatchSource:0}: Error finding container 7cbedf4e4e1a8c5f594d934ac6003c8cd51f487f3dbf610628804cc87c0d5c45: Status 404 returned error can't find the container with id 7cbedf4e4e1a8c5f594d934ac6003c8cd51f487f3dbf610628804cc87c0d5c45 Nov 01 05:31:22.654070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.654039 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:22.654146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.654104 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:22.654146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.654131 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:22.836374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.836351 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:31:22.837633 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:31:22.837611 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71eac8a_d6c1_4080_8244_5626c5b65195.slice/crio-f992d7be350ff3bdee04f344ec3a6a8f7452f169f5f5805429cf615f043e9134 WatchSource:0}: Error finding container f992d7be350ff3bdee04f344ec3a6a8f7452f169f5f5805429cf615f043e9134: Status 404 returned error can't find the container with id f992d7be350ff3bdee04f344ec3a6a8f7452f169f5f5805429cf615f043e9134 Nov 01 05:31:22.938948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.938907 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" event={"ID":"a71eac8a-d6c1-4080-8244-5626c5b65195","Type":"ContainerStarted","Data":"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32"} Nov 01 05:31:22.939674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.938982 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" event={"ID":"a71eac8a-d6c1-4080-8244-5626c5b65195","Type":"ContainerStarted","Data":"f992d7be350ff3bdee04f344ec3a6a8f7452f169f5f5805429cf615f043e9134"} Nov 01 05:31:22.939674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.939134 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:22.941281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.941231 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" event={"ID":"6270326e-b279-4e83-9876-ce4d9ab8af8f","Type":"ContainerStarted","Data":"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec"} Nov 01 05:31:22.941401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:22.941304 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" event={"ID":"6270326e-b279-4e83-9876-ce4d9ab8af8f","Type":"ContainerStarted","Data":"7cbedf4e4e1a8c5f594d934ac6003c8cd51f487f3dbf610628804cc87c0d5c45"} Nov 01 05:31:23.058956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.058854 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" podStartSLOduration=1.058841296 podStartE2EDuration="1.058841296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:23.05761359 +0000 UTC m=+1698.198202534" watchObservedRunningTime="2026-11-01 05:31:23.058841296 +0000 UTC m=+1698.199430214" Nov 01 05:31:23.181118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.181045 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podStartSLOduration=2.181027573 podStartE2EDuration="2.181027573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:23.179781358 +0000 UTC m=+1698.320370268" watchObservedRunningTime="2026-11-01 05:31:23.181027573 +0000 UTC m=+1698.321616523" Nov 01 05:31:23.224091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.224050 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:23.225989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.225964 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:23.226114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.226010 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:23.498865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.498792 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" containerID="cri-o://6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9" gracePeriod=25 Nov 01 05:31:23.608658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:23.608618 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" containerID="cri-o://f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594" gracePeriod=25 Nov 01 05:31:24.224105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.224067 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:24.224526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.224121 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:24.764673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.764630 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:31:24.770212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.770182 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.808003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.807971 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:31:24.862282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.856457 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-3bbeecafc00b\"" Nov 01 05:31:24.862282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.856574 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\"" Nov 01 05:31:24.862282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.856618 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-2e04ffa38ad1\"" Nov 01 05:31:24.866414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.866383 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-89znj\"" Nov 01 05:31:24.866648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.866464 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\"" Nov 01 05:31:24.866648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.866502 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\"" Nov 01 05:31:24.866648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.866559 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\"" Nov 01 05:31:24.872342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.872315 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.872491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.872401 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.872491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.872475 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.872588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.872523 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65knl\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.973945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.973909 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.974125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.973975 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.974125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.974016 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.974125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.974069 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65knl\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.974811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.974783 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.974924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.974817 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:24.976663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:24.976631 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:25.037423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:25.037335 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65knl\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:25.085441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:25.085397 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:25.226293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:25.223977 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:25.226293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:25.224022 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:25.497853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:25.497825 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:31:26.016299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.016223 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" event={"ID":"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8","Type":"ContainerStarted","Data":"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8"} Nov 01 05:31:26.016554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.016535 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" event={"ID":"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8","Type":"ContainerStarted","Data":"ce8bc6147bdff4beb17561f6ac40d74987997475bc26eb5f0b456ee63b0be32b"} Nov 01 05:31:26.016713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.016699 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:26.178279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.162095 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" podStartSLOduration=2.162074697 podStartE2EDuration="2.162074697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:26.157871561 +0000 UTC m=+1701.298460511" watchObservedRunningTime="2026-11-01 05:31:26.162074697 +0000 UTC m=+1701.302663621" Nov 01 05:31:26.224037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.223992 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:26.224213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.224045 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:26.480982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.480945 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8"] Nov 01 05:31:26.491519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.491492 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.542289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.541839 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8"] Nov 01 05:31:26.551280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.550566 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:31:26.551280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.551243 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:31:26.551511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.551343 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:31:26.551511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.551445 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:31:26.612278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.612181 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpdc\" (UniqueName: \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.612476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.612392 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.612476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.612458 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.722279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.719773 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.722279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.719860 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.722279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.719973 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpdc\" (UniqueName: \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.722279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.721215 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.737325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.737037 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.808003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.807965 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpdc\" (UniqueName: \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\") pod \"maas-api-revoke-sub-1232cfdc-gdsl8\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.816278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:26.815536 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:26.999330 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:26.998521 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:31:27.225869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:27.225834 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:27.226161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:27.226130 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:27.427608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:27.425056 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8"] Nov 01 05:31:28.081535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.081495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" event={"ID":"949bbcec-3a69-4f85-8c43-e6023903b0e9","Type":"ContainerStarted","Data":"ad52522251c7b0544a609c7569fa59f3e01483c8ea381d3ab37e1c230e3f986e"} Nov 01 05:31:28.082018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.081548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" event={"ID":"949bbcec-3a69-4f85-8c43-e6023903b0e9","Type":"ContainerStarted","Data":"ddf69499dc7ad76f5c0b49db34639f633fead3924818afccd7fd0a0dcddc1b92"} Nov 01 05:31:28.201540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.201484 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" podStartSLOduration=2.201470253 podStartE2EDuration="2.201470253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:28.199246232 +0000 UTC m=+1703.339835167" watchObservedRunningTime="2026-11-01 05:31:28.201470253 +0000 UTC m=+1703.342059171" Nov 01 05:31:28.224029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.223999 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:28.224187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.224055 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:28.621650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.620169 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:31:28.622687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.622643 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:31:28.679460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.670303 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:31:28.679460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:28.670603 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" containerID="cri-o://fca2cff5515d1aad0326356dabc6281c01c1759da88ff03227133f48a8ec1614" gracePeriod=30 Nov 01 05:31:29.095294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:29.095240 4425 generic.go:356] "Generic (PLEG): container finished" podID="949bbcec-3a69-4f85-8c43-e6023903b0e9" containerID="ad52522251c7b0544a609c7569fa59f3e01483c8ea381d3ab37e1c230e3f986e" exitCode=0 Nov 01 05:31:29.095294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:29.095286 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" event={"ID":"949bbcec-3a69-4f85-8c43-e6023903b0e9","Type":"ContainerDied","Data":"ad52522251c7b0544a609c7569fa59f3e01483c8ea381d3ab37e1c230e3f986e"} Nov 01 05:31:29.227651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:29.224171 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 05:31:29.227651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:29.224231 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 05:31:30.315563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.315531 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:30.319555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.319516 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:30.320017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.319993 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:30.320952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.320934 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:31:30.407273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.407156 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\" (UniqueName: \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\") pod \"949bbcec-3a69-4f85-8c43-e6023903b0e9\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " Nov 01 05:31:30.407273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.407218 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\") pod \"949bbcec-3a69-4f85-8c43-e6023903b0e9\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " Nov 01 05:31:30.407527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.407351 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\") pod \"949bbcec-3a69-4f85-8c43-e6023903b0e9\" (UID: \"949bbcec-3a69-4f85-8c43-e6023903b0e9\") " Nov 01 05:31:30.407669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.407637 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca" pod "949bbcec-3a69-4f85-8c43-e6023903b0e9" (UID: "949bbcec-3a69-4f85-8c43-e6023903b0e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:30.408050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.408000 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:30.409694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.409668 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca" pod "949bbcec-3a69-4f85-8c43-e6023903b0e9" (UID: "949bbcec-3a69-4f85-8c43-e6023903b0e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:31:30.409694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.409703 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc" pod "949bbcec-3a69-4f85-8c43-e6023903b0e9" (UID: "949bbcec-3a69-4f85-8c43-e6023903b0e9"). InnerVolumeSpecName "kube-api-access-vnpdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:30.509814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.509758 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpdc\" (UniqueName: \"kubernetes.io/projected/949bbcec-3a69-4f85-8c43-e6023903b0e9-kube-api-access-vnpdc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:30.509814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:30.509801 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949bbcec-3a69-4f85-8c43-e6023903b0e9-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:31.114819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:31.114786 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" Nov 01 05:31:31.114819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:31.114816 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8" event={"ID":"949bbcec-3a69-4f85-8c43-e6023903b0e9","Type":"ContainerDied","Data":"ddf69499dc7ad76f5c0b49db34639f633fead3924818afccd7fd0a0dcddc1b92"} Nov 01 05:31:31.115012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:31.114869 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf69499dc7ad76f5c0b49db34639f633fead3924818afccd7fd0a0dcddc1b92" Nov 01 05:31:33.985309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:33.985204 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:31:34.139168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.139136 4425 generic.go:356] "Generic (PLEG): container finished" podID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerID="fca2cff5515d1aad0326356dabc6281c01c1759da88ff03227133f48a8ec1614" exitCode=0 Nov 01 05:31:34.139368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.139206 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerDied","Data":"fca2cff5515d1aad0326356dabc6281c01c1759da88ff03227133f48a8ec1614"} Nov 01 05:31:34.139368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.139292 4425 scope.go:122] "RemoveContainer" containerID="ef7d4c7d9184114f7b8f79204b0f72039ee0ba85c205e6fd1353b62223d04ab7" Nov 01 05:31:34.347303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.347278 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:31:34.457550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457509 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457556 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457590 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457655 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457717 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457743 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.457800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457769 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.458076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457807 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.458076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457856 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:34.458076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457899 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\") pod \"62701486-39cc-4ea4-bbd3-5c33dd61e773\" (UID: \"62701486-39cc-4ea4-bbd3-5c33dd61e773\") " Nov 01 05:31:34.458076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.457920 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:34.458305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458117 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:34.458305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458165 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:34.458305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458169 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:34.458485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458449 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62701486-39cc-4ea4-bbd3-5c33dd61e773-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.458485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458478 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.458583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458493 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.458583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458508 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.458583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.458524 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.460131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.460104 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:34.460131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.460119 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:34.460623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.460607 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:34.460697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.460651 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj" pod "62701486-39cc-4ea4-bbd3-5c33dd61e773" (UID: "62701486-39cc-4ea4-bbd3-5c33dd61e773"). InnerVolumeSpecName "kube-api-access-psmlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:34.560326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.560221 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmlj\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-kube-api-access-psmlj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.560326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.560283 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.560326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.560295 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:34.560326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:34.560303 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62701486-39cc-4ea4-bbd3-5c33dd61e773-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:35.147706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:35.147673 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" event={"ID":"62701486-39cc-4ea4-bbd3-5c33dd61e773","Type":"ContainerDied","Data":"6717ea568623b8f504b31b6d8c1ea3b5364baff152b91fb40ccc70a8563bd757"} Nov 01 05:31:35.147706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:35.147701 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd" Nov 01 05:31:35.148173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:35.147731 4425 scope.go:122] "RemoveContainer" containerID="fca2cff5515d1aad0326356dabc6281c01c1759da88ff03227133f48a8ec1614" Nov 01 05:31:35.317666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:35.317632 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:31:35.368837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:35.368795 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gtrfd"] Nov 01 05:31:36.975265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:36.975226 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" path="/var/lib/kubelet/pods/62701486-39cc-4ea4-bbd3-5c33dd61e773/volumes" Nov 01 05:31:37.076476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:37.076447 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:31:38.142165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.142134 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 05:31:38.142923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.142897 4425 cpu_manager.go:414] "containerMap: removing container" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:31:38.142923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.142919 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:31:38.143162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.143035 4425 cpu_manager.go:414] "containerMap: removing container" podUID="949bbcec-3a69-4f85-8c43-e6023903b0e9" containerName="revoke-keys" Nov 01 05:31:38.143162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.143045 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949bbcec-3a69-4f85-8c43-e6023903b0e9" containerName="revoke-keys" Nov 01 05:31:38.143382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.143295 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="949bbcec-3a69-4f85-8c43-e6023903b0e9" containerName="revoke-keys" Nov 01 05:31:38.143382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.143313 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:31:38.143382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.143320 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:31:38.148790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.148772 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.205301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.204874 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-afc555-openshift-default-dockercfg-z5wx8\"" Nov 01 05:31:38.261188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.261157 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 05:31:38.308020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.307964 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308032 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308207 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308175 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308321 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308382 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308424 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308568 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wlb\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308612 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.308707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.308680 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.409887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409786 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.409887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409842 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.409887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409874 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409923 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409960 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.409992 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410034 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wlb\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410071 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410124 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410379 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410449 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410559 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410668 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.410823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.410725 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.412458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.412435 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.412691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.412672 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.491051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.491018 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wlb\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.491485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.491455 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\") pod \"e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:38.619385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.619346 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:31:38.619529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.619346 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:31:38.725363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.725277 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:31:38.725542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.725513 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" containerID="cri-o://bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad" gracePeriod=30 Nov 01 05:31:38.767426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:38.767396 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:39.206304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:39.206271 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 05:31:39.207524 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:31:39.207493 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764e847b_5005_4bc2_b50f_af824b3d68a6.slice/crio-abe1e223c5100dbe59d52d7c06fa41d7d22feece78c06975980d8cf1263423ad WatchSource:0}: Error finding container abe1e223c5100dbe59d52d7c06fa41d7d22feece78c06975980d8cf1263423ad: Status 404 returned error can't find the container with id abe1e223c5100dbe59d52d7c06fa41d7d22feece78c06975980d8cf1263423ad Nov 01 05:31:39.209793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:39.209764 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:39.209861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:39.209828 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:39.209902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:39.209858 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:31:40.216723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:40.216685 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerStarted","Data":"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca"} Nov 01 05:31:40.217153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:40.216744 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerStarted","Data":"abe1e223c5100dbe59d52d7c06fa41d7d22feece78c06975980d8cf1263423ad"} Nov 01 05:31:40.344211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:40.344158 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podStartSLOduration=2.344142808 podStartE2EDuration="2.344142808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:40.342749659 +0000 UTC m=+1715.483338579" watchObservedRunningTime="2026-11-01 05:31:40.344142808 +0000 UTC m=+1715.484731728" Nov 01 05:31:40.768194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:40.768160 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:40.774498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:40.774472 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:41.223697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:41.223663 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:41.224747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:41.224730 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:31:43.876446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.876415 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:31:43.906101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.905992 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:31:43.973969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.973935 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\") pod \"b88dc102-9c16-4192-8e40-4a3f81bfce54\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " Nov 01 05:31:43.974166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.973985 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\") pod \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " Nov 01 05:31:43.974166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\" (UniqueName: \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\") pod \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\" (UID: \"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d\") " Nov 01 05:31:43.974327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974244 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\" (UniqueName: \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\") pod \"b88dc102-9c16-4192-8e40-4a3f81bfce54\" (UID: \"b88dc102-9c16-4192-8e40-4a3f81bfce54\") " Nov 01 05:31:43.974434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974403 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume" pod "b88dc102-9c16-4192-8e40-4a3f81bfce54" (UID: "b88dc102-9c16-4192-8e40-4a3f81bfce54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:43.974505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974471 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume" pod "0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" (UID: "0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:43.974818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974791 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b88dc102-9c16-4192-8e40-4a3f81bfce54-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:43.974944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.974824 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:43.979366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.979335 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6" pod "0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" (UID: "0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d"). InnerVolumeSpecName "kube-api-access-2zjm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:43.979619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:43.979599 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8" pod "b88dc102-9c16-4192-8e40-4a3f81bfce54" (UID: "b88dc102-9c16-4192-8e40-4a3f81bfce54"). InnerVolumeSpecName "kube-api-access-wbxl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:44.004227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.004197 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:31:44.075318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075291 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075423 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075477 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075578 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075629 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075683 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075722 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075751 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.075811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075783 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\") pod \"975f14fd-353f-4421-94f0-44f8552ddee5\" (UID: \"975f14fd-353f-4421-94f0-44f8552ddee5\") " Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.075966 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076436 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076502 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxl8\" (UniqueName: \"kubernetes.io/projected/b88dc102-9c16-4192-8e40-4a3f81bfce54-kube-api-access-wbxl8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076530 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076546 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjm6\" (UniqueName: \"kubernetes.io/projected/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d-kube-api-access-2zjm6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076583 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:31:44.076729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076679 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:44.077193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.076774 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:44.077952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.077925 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:31:44.077952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.077935 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:44.078096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.077956 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "kube-api-access-h6g5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:31:44.078096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.078021 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy" pod "975f14fd-353f-4421-94f0-44f8552ddee5" (UID: "975f14fd-353f-4421-94f0-44f8552ddee5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:31:44.177412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177380 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177416 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177430 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975f14fd-353f-4421-94f0-44f8552ddee5-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177438 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177447 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177456 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6g5t\" (UniqueName: \"kubernetes.io/projected/975f14fd-353f-4421-94f0-44f8552ddee5-kube-api-access-h6g5t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177465 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975f14fd-353f-4421-94f0-44f8552ddee5-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.177670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.177473 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975f14fd-353f-4421-94f0-44f8552ddee5-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:31:44.245635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.245598 4425 generic.go:356] "Generic (PLEG): container finished" podID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerID="f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594" exitCode=0 Nov 01 05:31:44.245811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.245651 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" Nov 01 05:31:44.245811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.245686 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" event={"ID":"b88dc102-9c16-4192-8e40-4a3f81bfce54","Type":"ContainerDied","Data":"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594"} Nov 01 05:31:44.245811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.245740 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc" event={"ID":"b88dc102-9c16-4192-8e40-4a3f81bfce54","Type":"ContainerDied","Data":"56f2ae6301d64d8d7494337a5c2b29f5a7cf5ae9c724f81eb45a4d6299df70b7"} Nov 01 05:31:44.245811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.245767 4425 scope.go:122] "RemoveContainer" containerID="f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594" Nov 01 05:31:44.247243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.247213 4425 generic.go:356] "Generic (PLEG): container finished" podID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerID="6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9" exitCode=0 Nov 01 05:31:44.247394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.247242 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" event={"ID":"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d","Type":"ContainerDied","Data":"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9"} Nov 01 05:31:44.247394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.247307 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" Nov 01 05:31:44.247394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.247324 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l" event={"ID":"0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d","Type":"ContainerDied","Data":"0e978f5b3fd5c6d11aaa1c67a9136b5084488c1bbc7b80cb091666d5cc190ce9"} Nov 01 05:31:44.249604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.249576 4425 generic.go:356] "Generic (PLEG): container finished" podID="975f14fd-353f-4421-94f0-44f8552ddee5" containerID="bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad" exitCode=0 Nov 01 05:31:44.249728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.249629 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" Nov 01 05:31:44.249728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.249635 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerDied","Data":"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad"} Nov 01 05:31:44.249728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.249695 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp" event={"ID":"975f14fd-353f-4421-94f0-44f8552ddee5","Type":"ContainerDied","Data":"0803a68a785899ef5d945e48356c18946c06a167cd3fbf3e15eec5a1c83c6621"} Nov 01 05:31:44.258236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.258214 4425 scope.go:122] "RemoveContainer" containerID="f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594" Nov 01 05:31:44.258610 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:44.258577 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594\": container with ID starting with f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594 not found: ID does not exist" containerID="f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594" Nov 01 05:31:44.258717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.258620 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594"} err="failed to get container status \"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594\": rpc error: code = NotFound desc = could not find container \"f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594\": container with ID starting with f96ae3bd1c2685f979b573f318b6bce5b0f9f63be6d33c5069830f1b4f159594 not found: ID does not exist" Nov 01 05:31:44.258717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.258643 4425 scope.go:122] "RemoveContainer" containerID="6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9" Nov 01 05:31:44.268312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.268295 4425 scope.go:122] "RemoveContainer" containerID="6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9" Nov 01 05:31:44.268615 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:44.268597 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9\": container with ID starting with 6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9 not found: ID does not exist" containerID="6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9" Nov 01 05:31:44.268671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.268623 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9"} err="failed to get container status \"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9\": rpc error: code = NotFound desc = could not find container \"6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9\": container with ID starting with 6ca8fa942b1e57836c98cce4221eefc28491ff599b828d698be1b4363e76fba9 not found: ID does not exist" Nov 01 05:31:44.268671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.268641 4425 scope.go:122] "RemoveContainer" containerID="bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad" Nov 01 05:31:44.277767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.277743 4425 scope.go:122] "RemoveContainer" containerID="31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03" Nov 01 05:31:44.288165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.288137 4425 scope.go:122] "RemoveContainer" containerID="bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad" Nov 01 05:31:44.288537 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:44.288514 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad\": container with ID starting with bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad not found: ID does not exist" containerID="bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad" Nov 01 05:31:44.288641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.288545 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad"} err="failed to get container status \"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad\": rpc error: code = NotFound desc = could not find container \"bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad\": container with ID starting with bdc450d5e4b0eda7f7fe393740920a9a9aa56cfdeb27a5bd31f596b28a6a02ad not found: ID does not exist" Nov 01 05:31:44.288641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.288569 4425 scope.go:122] "RemoveContainer" containerID="31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03" Nov 01 05:31:44.288842 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:44.288820 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03\": container with ID starting with 31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03 not found: ID does not exist" containerID="31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03" Nov 01 05:31:44.288950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.288848 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03"} err="failed to get container status \"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03\": rpc error: code = NotFound desc = could not find container \"31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03\": container with ID starting with 31a84fce21884f13f508fbb6925ea2eef27bdab60eb11fca5017c9406a44ad03 not found: ID does not exist" Nov 01 05:31:44.404159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.404078 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:31:44.409644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.409617 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-918b8d-7767795f95-4p84l"] Nov 01 05:31:44.522807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.522781 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:31:44.532144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.532117 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-918b8d-openshift-default-6c7bccc7d5-zs5bp"] Nov 01 05:31:44.643551 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.643518 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:31:44.649896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.649859 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-918b8d-c7ffd76dc-l5npc"] Nov 01 05:31:44.973905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.973871 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" path="/var/lib/kubelet/pods/0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d/volumes" Nov 01 05:31:44.974386 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.974372 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" path="/var/lib/kubelet/pods/975f14fd-353f-4421-94f0-44f8552ddee5/volumes" Nov 01 05:31:44.974769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:31:44.974757 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" path="/var/lib/kubelet/pods/b88dc102-9c16-4192-8e40-4a3f81bfce54/volumes" Nov 01 05:31:56.953318 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:31:56.953245 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:32:06.418811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:06.418772 4425 scope.go:122] "RemoveContainer" containerID="466ba23dba92653ba4aabbf9bd3a35e6222cd6fd52a2f93c5b60591879da0680" Nov 01 05:32:26.947456 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:32:26.947410 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:32:30.617420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:30.617375 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:32:30.617873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:30.617569 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" containerID="cri-o://ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b" gracePeriod=30 Nov 01 05:32:30.702700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:30.702663 4425 patch_prober.go:29] interesting pod/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:32:30.702863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:30.702729 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:32:35.903787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.903758 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:32:35.950327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950293 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950336 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950373 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950414 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950449 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950491 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950539 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950617 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950710 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:32:35.950787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950717 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\") pod \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\" (UID: \"9bb211b3-12af-4e82-9bc5-6a0478a55fbe\") " Nov 01 05:32:35.950787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950734 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:32:35.951128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950844 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:32:35.951128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.950918 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:32:35.951242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.951164 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:32:35.951242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.951212 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:35.951242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.951227 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:35.951242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.951236 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:35.951242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.951246 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:35.952860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.952832 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:32:35.953165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.953145 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:32:35.953406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.953389 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:32:35.953461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:35.953400 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p" pod "9bb211b3-12af-4e82-9bc5-6a0478a55fbe" (UID: "9bb211b3-12af-4e82-9bc5-6a0478a55fbe"). InnerVolumeSpecName "kube-api-access-77z2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:32:36.052734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.052691 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:36.052734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.052723 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:36.052734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.052735 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77z2p\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-kube-api-access-77z2p\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:36.052734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.052746 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:36.053016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.052756 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb211b3-12af-4e82-9bc5-6a0478a55fbe-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:32:36.603425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.603383 4425 generic.go:356] "Generic (PLEG): container finished" podID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerID="ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b" exitCode=0 Nov 01 05:32:36.603642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.603445 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" Nov 01 05:32:36.603642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.603467 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerDied","Data":"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b"} Nov 01 05:32:36.603642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.603525 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9" event={"ID":"9bb211b3-12af-4e82-9bc5-6a0478a55fbe","Type":"ContainerDied","Data":"482928ee4f54668c70eb3e54f0566c42be3f5e4f32764166808a0ff1b8750df0"} Nov 01 05:32:36.603642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.603547 4425 scope.go:122] "RemoveContainer" containerID="ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b" Nov 01 05:32:36.614463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.614445 4425 scope.go:122] "RemoveContainer" containerID="b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0" Nov 01 05:32:36.623414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.623390 4425 scope.go:122] "RemoveContainer" containerID="ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b" Nov 01 05:32:36.623707 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:32:36.623688 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b\": container with ID starting with ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b not found: ID does not exist" containerID="ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b" Nov 01 05:32:36.623772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.623715 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b"} err="failed to get container status \"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b\": rpc error: code = NotFound desc = could not find container \"ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b\": container with ID starting with ddd891acd9f6e2e5b632d9708527a7ba7364245ad08734360eaf70926ea9520b not found: ID does not exist" Nov 01 05:32:36.623772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.623732 4425 scope.go:122] "RemoveContainer" containerID="b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0" Nov 01 05:32:36.624001 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:32:36.623965 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0\": container with ID starting with b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0 not found: ID does not exist" containerID="b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0" Nov 01 05:32:36.624001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.623991 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0"} err="failed to get container status \"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0\": rpc error: code = NotFound desc = could not find container \"b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0\": container with ID starting with b8c1a5a5bd87fcf14833b16baba8f69b83ecc00982e3af693580ee1513f16fd0 not found: ID does not exist" Nov 01 05:32:36.823056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.823023 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:32:36.873897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.873822 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca7a55-openshift-default-5fbc7f68d5-cx2k9"] Nov 01 05:32:36.973606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:36.973572 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" path="/var/lib/kubelet/pods/9bb211b3-12af-4e82-9bc5-6a0478a55fbe/volumes" Nov 01 05:32:41.407811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.407770 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:32:41.408552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408531 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408553 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408578 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408585 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408613 4425 cpu_manager.go:414] "containerMap: removing container" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408620 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408639 4425 cpu_manager.go:414] "containerMap: removing container" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:32:41.408661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408647 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62701486-39cc-4ea4-bbd3-5c33dd61e773" containerName="istio-proxy" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408683 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408691 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408825 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408837 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408852 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88dc102-9c16-4192-8e40-4a3f81bfce54" containerName="payload-pre-processing" Nov 01 05:32:41.409011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.408862 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9be87d-74f5-4d9d-8e85-3c8ff4e6651d" containerName="payload-processing" Nov 01 05:32:41.414009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.413987 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.517480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517428 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.517682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517587 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.517749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517669 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517822 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517868 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517893 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517964 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5th6\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.517997 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.518031 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.523265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.523217 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:32:41.586461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.586422 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ec52ff90-openshift-default-dockercfg-6kj68\"" Nov 01 05:32:41.618689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.618657 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619244 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619367 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619484 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619531 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619557 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619632 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5th6\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619633 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619668 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619698 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.619901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.619876 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.620125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.620104 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.620382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.620362 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.620697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.620670 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.621620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.621600 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.623787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.623763 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.682331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.682236 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.682481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.682457 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5th6\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\") pod \"e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:41.726918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:41.726888 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:42.188930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.188898 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:32:42.189059 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:32:42.189030 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e325088_1cc0_437a_92d6_2b0afff19b6f.slice/crio-353146f152d939d80964f2e929e26d4558b358b7eda2d4d7cd3257cb61f8ce7e WatchSource:0}: Error finding container 353146f152d939d80964f2e929e26d4558b358b7eda2d4d7cd3257cb61f8ce7e: Status 404 returned error can't find the container with id 353146f152d939d80964f2e929e26d4558b358b7eda2d4d7cd3257cb61f8ce7e Nov 01 05:32:42.191397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.191363 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:42.191475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.191427 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:42.191475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.191460 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:42.673360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.673312 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerStarted","Data":"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634"} Nov 01 05:32:42.673871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.673400 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerStarted","Data":"353146f152d939d80964f2e929e26d4558b358b7eda2d4d7cd3257cb61f8ce7e"} Nov 01 05:32:42.727332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.727302 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:42.796871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:42.796821 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podStartSLOduration=1.7968079559999999 podStartE2EDuration="1.796807956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:32:42.795116351 +0000 UTC m=+1777.935705300" watchObservedRunningTime="2026-11-01 05:32:42.796807956 +0000 UTC m=+1777.937396874" Nov 01 05:32:43.733202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:43.733170 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:44.685892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:44.685860 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:44.686813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:44.686795 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:32:48.494536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.494499 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:32:48.495290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495269 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:48.495401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495291 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:48.495401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495320 4425 cpu_manager.go:414] "containerMap: removing container" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:48.495401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495328 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:48.495558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495503 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="975f14fd-353f-4421-94f0-44f8552ddee5" containerName="istio-proxy" Nov 01 05:32:48.495558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.495515 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb211b3-12af-4e82-9bc5-6a0478a55fbe" containerName="istio-proxy" Nov 01 05:32:48.499109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.499089 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.558671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.558631 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:32:48.611616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.611566 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ec52ff90-dockercfg-bspqr\"" Nov 01 05:32:48.611837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.611701 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ec52ff90\"" Nov 01 05:32:48.634711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.634670 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.635021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.634995 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgl5j\" (UniqueName: \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.667027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.666980 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:32:48.675238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.675209 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:48.702528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.702497 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:32:48.736085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.736046 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgl5j\" (UniqueName: \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.736301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.736147 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.737319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.737293 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.800102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.800075 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgl5j\" (UniqueName: \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\") pod \"payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.816406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.816381 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:48.837653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.837624 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44b29\" (UniqueName: \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:48.837791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.837739 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:48.939179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.939085 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44b29\" (UniqueName: \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:48.939179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.939157 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:48.939890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:48.939867 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:49.001428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.001339 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44b29\" (UniqueName: \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\") pod \"payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:49.198009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.197981 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:32:49.199094 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:32:49.199069 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd215d476_bd01_4f54_b4c3_30a3e6a96c44.slice/crio-0175e96b1ebe9d382d136a74b96560f796c3243121d9dde1307d682c032515d5 WatchSource:0}: Error finding container 0175e96b1ebe9d382d136a74b96560f796c3243121d9dde1307d682c032515d5: Status 404 returned error can't find the container with id 0175e96b1ebe9d382d136a74b96560f796c3243121d9dde1307d682c032515d5 Nov 01 05:32:49.289355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.289327 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:49.668748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.668708 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:32:49.669186 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:32:49.669145 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fc3782_aab8_415a_ac6d_0f18e05a3ee5.slice/crio-bc105ab93b366f81de800a8b77123e26ddf26b2c2c759ffc2822f71030fab4ef WatchSource:0}: Error finding container bc105ab93b366f81de800a8b77123e26ddf26b2c2c759ffc2822f71030fab4ef: Status 404 returned error can't find the container with id bc105ab93b366f81de800a8b77123e26ddf26b2c2c759ffc2822f71030fab4ef Nov 01 05:32:49.723156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.723116 4425 generic.go:356] "Generic (PLEG): container finished" podID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerID="8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca" exitCode=0 Nov 01 05:32:49.723302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.723185 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerDied","Data":"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca"} Nov 01 05:32:49.723639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.723615 4425 scope.go:122] "RemoveContainer" containerID="8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca" Nov 01 05:32:49.725060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725029 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" event={"ID":"d215d476-bd01-4f54-b4c3-30a3e6a96c44","Type":"ContainerStarted","Data":"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e"} Nov 01 05:32:49.725201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725092 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" event={"ID":"d215d476-bd01-4f54-b4c3-30a3e6a96c44","Type":"ContainerStarted","Data":"0175e96b1ebe9d382d136a74b96560f796c3243121d9dde1307d682c032515d5"} Nov 01 05:32:49.725201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725134 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:32:49.725361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725191 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:49.725361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725288 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:49.725361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.725336 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:32:49.726558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.726538 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" event={"ID":"50fc3782-aab8-415a-ac6d-0f18e05a3ee5","Type":"ContainerStarted","Data":"bc105ab93b366f81de800a8b77123e26ddf26b2c2c759ffc2822f71030fab4ef"} Nov 01 05:32:49.965726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:49.965623 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" podStartSLOduration=1.965606042 podStartE2EDuration="1.965606042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:32:49.963117588 +0000 UTC m=+1785.103706504" watchObservedRunningTime="2026-11-01 05:32:49.965606042 +0000 UTC m=+1785.106194961" Nov 01 05:32:50.734392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.734347 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" event={"ID":"50fc3782-aab8-415a-ac6d-0f18e05a3ee5","Type":"ContainerStarted","Data":"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3"} Nov 01 05:32:50.734847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.734424 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:32:50.736587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.736563 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerStarted","Data":"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0"} Nov 01 05:32:50.770801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.770767 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:32:50.771048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.771008 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:50.771202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.771070 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:50.875360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:50.875305 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" podStartSLOduration=2.875290691 podStartE2EDuration="2.875290691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:32:50.874336526 +0000 UTC m=+1786.014925446" watchObservedRunningTime="2026-11-01 05:32:50.875290691 +0000 UTC m=+1786.015879607" Nov 01 05:32:51.768418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:51.768384 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:51.768961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:51.768444 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:52.768756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:52.768719 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:52.769303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:52.768784 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:53.768638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:53.768601 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:53.768818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:53.768655 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:54.768802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:54.768759 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:54.768997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:54.768856 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:55.768392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:55.768349 4425 patch_prober.go:29] interesting pod/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 05:32:55.768589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:55.768437 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 05:32:56.224246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:56.224211 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:32:56.772826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:56.772795 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:32:56.802302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:32:56.802277 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 05:32:56.947788 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:32:56.947754 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:33:00.740032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:00.740004 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:33:01.768334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:01.768303 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:33:06.475313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:06.475281 4425 scope.go:122] "RemoveContainer" containerID="7bad98cbf61b3d733c4f231494297968d506467503856a9a28940fbb9d468f42" Nov 01 05:33:06.484907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:06.484881 4425 scope.go:122] "RemoveContainer" containerID="5d2bf44d3c04ca5e50dcf2aa263f0ca821d3749c858599935db5bac5bb44fe34" Nov 01 05:33:26.948126 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:33:26.948080 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:33:30.536785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.536751 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 05:33:30.541154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.541131 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.598282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.598219 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 05:33:30.643018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.642973 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjl7w\" (UniqueName: \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.644494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.644470 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.645501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.645133 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-1c4191-dockercfg-7ggt8\"" Nov 01 05:33:30.645677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.645272 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-1c4191\"" Nov 01 05:33:30.699830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.699800 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 05:33:30.704435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.704412 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.745979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.745951 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 05:33:30.746553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.746535 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjl7w\" (UniqueName: \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.746627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.746578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.746715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.746696 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbq5\" (UniqueName: \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.746813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.746793 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.747431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.747407 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.808774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.808708 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjl7w\" (UniqueName: \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\") pod \"payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.848334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.848302 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.848521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.848504 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbq5\" (UniqueName: \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.849036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.849018 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:30.855026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.855007 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:30.911323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:30.911293 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbq5\" (UniqueName: \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\") pod \"payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:31.029006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:31.028974 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:31.236952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:31.236922 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 05:33:31.237574 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:33:31.237548 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416a0576_85f9_414d_ab4a_1b1f4ed9dae6.slice/crio-decc0541b6394b0002c9a0e606d9aa184ea4f9761d4e1d9dc7aa765670c0c5c0 WatchSource:0}: Error finding container decc0541b6394b0002c9a0e606d9aa184ea4f9761d4e1d9dc7aa765670c0c5c0: Status 404 returned error can't find the container with id decc0541b6394b0002c9a0e606d9aa184ea4f9761d4e1d9dc7aa765670c0c5c0 Nov 01 05:33:31.413863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:31.413827 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 05:33:31.418369 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:33:31.418326 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866fb760_d87d_4dbe_8673_a498b9fad4ee.slice/crio-250a3634c068dc3da527d1fe599acf50c530e2a028f26e3f546d374805db3a33 WatchSource:0}: Error finding container 250a3634c068dc3da527d1fe599acf50c530e2a028f26e3f546d374805db3a33: Status 404 returned error can't find the container with id 250a3634c068dc3da527d1fe599acf50c530e2a028f26e3f546d374805db3a33 Nov 01 05:33:32.018163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.018122 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" event={"ID":"416a0576-85f9-414d-ab4a-1b1f4ed9dae6","Type":"ContainerStarted","Data":"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52"} Nov 01 05:33:32.018658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.018185 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" event={"ID":"416a0576-85f9-414d-ab4a-1b1f4ed9dae6","Type":"ContainerStarted","Data":"decc0541b6394b0002c9a0e606d9aa184ea4f9761d4e1d9dc7aa765670c0c5c0"} Nov 01 05:33:32.018658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.018214 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:32.019780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.019759 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" event={"ID":"866fb760-d87d-4dbe-8673-a498b9fad4ee","Type":"ContainerStarted","Data":"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a"} Nov 01 05:33:32.019852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.019799 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" event={"ID":"866fb760-d87d-4dbe-8673-a498b9fad4ee","Type":"ContainerStarted","Data":"250a3634c068dc3da527d1fe599acf50c530e2a028f26e3f546d374805db3a33"} Nov 01 05:33:32.019852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.019826 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:32.137282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.137213 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" podStartSLOduration=2.137196 podStartE2EDuration="2.137196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:33:32.135016096 +0000 UTC m=+1827.275605018" watchObservedRunningTime="2026-11-01 05:33:32.137196 +0000 UTC m=+1827.277784919" Nov 01 05:33:32.258401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:32.258335 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" podStartSLOduration=2.258310329 podStartE2EDuration="2.258310329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:33:32.254646582 +0000 UTC m=+1827.395235501" watchObservedRunningTime="2026-11-01 05:33:32.258310329 +0000 UTC m=+1827.398899249" Nov 01 05:33:43.029827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:43.029800 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 05:33:43.030244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:43.029869 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 05:33:55.207443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.207408 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerID="c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634" exitCode=0 Nov 01 05:33:55.207926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.207484 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerDied","Data":"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634"} Nov 01 05:33:55.207926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.207896 4425 scope.go:122] "RemoveContainer" containerID="c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634" Nov 01 05:33:55.209415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.209383 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:33:55.209525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.209471 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:33:55.209525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:55.209519 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:33:56.218700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:56.218647 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerStarted","Data":"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05"} Nov 01 05:33:56.727593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:56.727558 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:33:56.728962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:56.728937 4425 patch_prober.go:29] interesting pod/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 05:33:56.729093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:56.728995 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 05:33:56.972245 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:33:56.972209 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:33:57.728062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:57.728027 4425 patch_prober.go:29] interesting pod/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 05:33:57.728584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:57.728097 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 05:33:58.728075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:58.728039 4425 patch_prober.go:29] interesting pod/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 05:33:58.728583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:58.728097 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 05:33:59.686452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:59.686416 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:33:59.727743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:59.727713 4425 patch_prober.go:29] interesting pod/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 05:33:59.727903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:33:59.727766 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 05:34:00.727634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:00.727605 4425 patch_prober.go:29] interesting pod/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 05:34:00.728070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:00.727672 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 05:34:01.732397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:01.732358 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:34:01.733663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:01.733636 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:34:22.976345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:22.976307 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:34:22.976794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:22.976577 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerName="discovery" containerID="cri-o://96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e" gracePeriod=30 Nov 01 05:34:23.275179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.275152 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:34:23.403971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.403935 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404049 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404093 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404123 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404209 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404307 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.404518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.404419 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\") pod \"afd64301-b5b2-4179-82f7-63f7fc574ab2\" (UID: \"afd64301-b5b2-4179-82f7-63f7fc574ab2\") " Nov 01 05:34:23.405268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.405197 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:34:23.407506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.407474 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:34:23.407732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.407688 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:34:23.407849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.407767 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:34:23.407849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.407789 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "kube-api-access-jlpr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:34:23.407849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.407793 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:34:23.408191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.408172 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token" pod "afd64301-b5b2-4179-82f7-63f7fc574ab2" (UID: "afd64301-b5b2-4179-82f7-63f7fc574ab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:34:23.417718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.417691 4425 generic.go:356] "Generic (PLEG): container finished" podID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerID="96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e" exitCode=0 Nov 01 05:34:23.417823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.417747 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" Nov 01 05:34:23.417823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.417777 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" event={"ID":"afd64301-b5b2-4179-82f7-63f7fc574ab2","Type":"ContainerDied","Data":"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e"} Nov 01 05:34:23.417974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.417853 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l" event={"ID":"afd64301-b5b2-4179-82f7-63f7fc574ab2","Type":"ContainerDied","Data":"30981fd7d54a11bc78bfd82a213eb83779a909d92abaf510e8cf01003fdf2270"} Nov 01 05:34:23.417974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.417885 4425 scope.go:122] "RemoveContainer" containerID="96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e" Nov 01 05:34:23.438366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.438228 4425 scope.go:122] "RemoveContainer" containerID="96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e" Nov 01 05:34:23.438605 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:34:23.438587 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e\": container with ID starting with 96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e not found: ID does not exist" containerID="96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e" Nov 01 05:34:23.438651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.438616 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e"} err="failed to get container status \"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e\": rpc error: code = NotFound desc = could not find container \"96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e\": container with ID starting with 96b973e03bc98e15aff9f54a20e4fd304a0e9c37041119d0124d2f3234ae456e not found: ID does not exist" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508365 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508391 4425 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-ca-configmap\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508403 4425 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd64301-b5b2-4179-82f7-63f7fc574ab2-local-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508411 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpr7\" (UniqueName: \"kubernetes.io/projected/afd64301-b5b2-4179-82f7-63f7fc574ab2-kube-api-access-jlpr7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508420 4425 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-kubeconfig\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508428 4425 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-istio-csr-dns-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.508442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.508437 4425 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd64301-b5b2-4179-82f7-63f7fc574ab2-cacerts\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:23.632802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.632767 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:34:23.637401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:23.637373 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gwt5l"] Nov 01 05:34:24.973913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:24.973878 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" path="/var/lib/kubelet/pods/afd64301-b5b2-4179-82f7-63f7fc574ab2/volumes" Nov 01 05:34:26.948815 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:34:26.948776 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:34:32.993785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:32.993751 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7"] Nov 01 05:34:32.994350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:32.994335 4425 cpu_manager.go:414] "containerMap: removing container" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerName="discovery" Nov 01 05:34:32.994405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:32.994350 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerName="discovery" Nov 01 05:34:32.994465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:32.994456 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd64301-b5b2-4179-82f7-63f7fc574ab2" containerName="discovery" Nov 01 05:34:32.997928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:32.997912 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.056458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.056426 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7"] Nov 01 05:34:33.102678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.102639 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:34:33.102678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.102639 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:34:33.102678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.102683 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:34:33.102938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.102757 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:34:33.118351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.118320 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.118535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.118462 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9692\" (UniqueName: \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.118603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.118544 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.219211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.219174 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.219452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.219301 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.219452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.219365 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9692\" (UniqueName: \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.220041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.220019 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.221836 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.221814 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.286341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.285912 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9692\" (UniqueName: \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\") pod \"maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.311577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.311547 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:33.695704 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:34:33.695658 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb41c0ca_11e1_4c31_bdf0_b590e389cbb9.slice/crio-259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9 WatchSource:0}: Error finding container 259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9: Status 404 returned error can't find the container with id 259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9 Nov 01 05:34:33.695982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:33.695957 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7"] Nov 01 05:34:34.488177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:34.488138 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" event={"ID":"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9","Type":"ContainerStarted","Data":"decf6b535fe92defe8e9d5afd5079a30bf98328a4564a62cffacd90d0ffadb26"} Nov 01 05:34:34.488709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:34.488199 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" event={"ID":"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9","Type":"ContainerStarted","Data":"259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9"} Nov 01 05:34:34.608294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:34.608227 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" podStartSLOduration=2.608213351 podStartE2EDuration="2.608213351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:34:34.606221302 +0000 UTC m=+1889.746810248" watchObservedRunningTime="2026-11-01 05:34:34.608213351 +0000 UTC m=+1889.748802269" Nov 01 05:34:35.495727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:35.495689 4425 generic.go:356] "Generic (PLEG): container finished" podID="cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" containerID="decf6b535fe92defe8e9d5afd5079a30bf98328a4564a62cffacd90d0ffadb26" exitCode=0 Nov 01 05:34:35.496153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:35.495739 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" event={"ID":"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9","Type":"ContainerDied","Data":"decf6b535fe92defe8e9d5afd5079a30bf98328a4564a62cffacd90d0ffadb26"} Nov 01 05:34:36.654803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.654774 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:36.760699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.760614 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\") pod \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " Nov 01 05:34:36.760851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.760751 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\") pod \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " Nov 01 05:34:36.760851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.760792 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\" (UniqueName: \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\") pod \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\" (UID: \"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9\") " Nov 01 05:34:36.761104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.761077 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca" pod "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" (UID: "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:34:36.763057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.763029 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca" pod "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" (UID: "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:34:36.763227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.763201 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692" pod "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" (UID: "cb41c0ca-11e1-4c31-bdf0-b590e389cbb9"). InnerVolumeSpecName "kube-api-access-m9692". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:34:36.862837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.862798 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:36.862837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.862831 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:36.862837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:36.862842 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9692\" (UniqueName: \"kubernetes.io/projected/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9-kube-api-access-m9692\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:37.510400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:37.510364 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" Nov 01 05:34:37.510400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:37.510376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7" event={"ID":"cb41c0ca-11e1-4c31-bdf0-b590e389cbb9","Type":"ContainerDied","Data":"259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9"} Nov 01 05:34:37.510629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:37.510430 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259c91a59c45ca91be2878a5477f5bcb28b9a0a8a0d6a68647a0af08bcde4ad9" Nov 01 05:34:44.503716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:44.503683 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:34:44.611901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:44.611869 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:34:49.504862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.504821 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" podUID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" containerName="payload-processing" containerID="cri-o://4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3" gracePeriod=25 Nov 01 05:34:49.613090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.613042 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" podUID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" containerName="payload-pre-processing" containerID="cri-o://3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e" gracePeriod=25 Nov 01 05:34:49.800516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.800485 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:34:49.874394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.874369 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:34:49.917530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.917499 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\") pod \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " Nov 01 05:34:49.917792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.917689 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\" (UniqueName: \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\") pod \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\" (UID: \"50fc3782-aab8-415a-ac6d-0f18e05a3ee5\") " Nov 01 05:34:49.919133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.918914 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume" pod "50fc3782-aab8-415a-ac6d-0f18e05a3ee5" (UID: "50fc3782-aab8-415a-ac6d-0f18e05a3ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:34:49.919486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.919448 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:49.930189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:49.924795 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29" pod "50fc3782-aab8-415a-ac6d-0f18e05a3ee5" (UID: "50fc3782-aab8-415a-ac6d-0f18e05a3ee5"). InnerVolumeSpecName "kube-api-access-44b29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:34:50.020083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.020004 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\" (UniqueName: \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\") pod \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " Nov 01 05:34:50.020083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.020074 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\") pod \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\" (UID: \"d215d476-bd01-4f54-b4c3-30a3e6a96c44\") " Nov 01 05:34:50.020490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.020458 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume" pod "d215d476-bd01-4f54-b4c3-30a3e6a96c44" (UID: "d215d476-bd01-4f54-b4c3-30a3e6a96c44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:34:50.020611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.020515 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44b29\" (UniqueName: \"kubernetes.io/projected/50fc3782-aab8-415a-ac6d-0f18e05a3ee5-kube-api-access-44b29\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:50.022358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.022332 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j" pod "d215d476-bd01-4f54-b4c3-30a3e6a96c44" (UID: "d215d476-bd01-4f54-b4c3-30a3e6a96c44"). InnerVolumeSpecName "kube-api-access-kgl5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:34:50.121267 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.121225 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgl5j\" (UniqueName: \"kubernetes.io/projected/d215d476-bd01-4f54-b4c3-30a3e6a96c44-kube-api-access-kgl5j\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:50.121455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.121289 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d215d476-bd01-4f54-b4c3-30a3e6a96c44-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:34:50.592157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.592118 4425 generic.go:356] "Generic (PLEG): container finished" podID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" containerID="3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e" exitCode=0 Nov 01 05:34:50.592723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.592192 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" Nov 01 05:34:50.592723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.592201 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" event={"ID":"d215d476-bd01-4f54-b4c3-30a3e6a96c44","Type":"ContainerDied","Data":"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e"} Nov 01 05:34:50.592723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.592275 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x" event={"ID":"d215d476-bd01-4f54-b4c3-30a3e6a96c44","Type":"ContainerDied","Data":"0175e96b1ebe9d382d136a74b96560f796c3243121d9dde1307d682c032515d5"} Nov 01 05:34:50.592723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.592310 4425 scope.go:122] "RemoveContainer" containerID="3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e" Nov 01 05:34:50.593761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.593737 4425 generic.go:356] "Generic (PLEG): container finished" podID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" containerID="4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3" exitCode=0 Nov 01 05:34:50.593841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.593803 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" Nov 01 05:34:50.593912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.593803 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" event={"ID":"50fc3782-aab8-415a-ac6d-0f18e05a3ee5","Type":"ContainerDied","Data":"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3"} Nov 01 05:34:50.593912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.593909 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p" event={"ID":"50fc3782-aab8-415a-ac6d-0f18e05a3ee5","Type":"ContainerDied","Data":"bc105ab93b366f81de800a8b77123e26ddf26b2c2c759ffc2822f71030fab4ef"} Nov 01 05:34:50.604246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.604170 4425 scope.go:122] "RemoveContainer" containerID="3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e" Nov 01 05:34:50.604598 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:34:50.604565 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e\": container with ID starting with 3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e not found: ID does not exist" containerID="3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e" Nov 01 05:34:50.604713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.604674 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e"} err="failed to get container status \"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e\": rpc error: code = NotFound desc = could not find container \"3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e\": container with ID starting with 3a5f7733700f94c5207f2ced06e057ffcec21aaa4c01de53d2950e14805b209e not found: ID does not exist" Nov 01 05:34:50.604781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.604713 4425 scope.go:122] "RemoveContainer" containerID="4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3" Nov 01 05:34:50.614188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.614171 4425 scope.go:122] "RemoveContainer" containerID="4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3" Nov 01 05:34:50.614467 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:34:50.614450 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3\": container with ID starting with 4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3 not found: ID does not exist" containerID="4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3" Nov 01 05:34:50.614525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.614472 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3"} err="failed to get container status \"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3\": rpc error: code = NotFound desc = could not find container \"4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3\": container with ID starting with 4ddb3a5d8bb8ea0a1f75714cbe61da268257b3f1486adab003d4c4d9d6fd72f3 not found: ID does not exist" Nov 01 05:34:50.761838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.761799 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:34:50.767614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.767586 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec52ff90-7c684dbfb7-k8g5x"] Nov 01 05:34:50.879030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.878992 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:34:50.883405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.883374 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec52ff90-6667b4d97f-26q5p"] Nov 01 05:34:50.973304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.973237 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" path="/var/lib/kubelet/pods/50fc3782-aab8-415a-ac6d-0f18e05a3ee5/volumes" Nov 01 05:34:50.973710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:34:50.973696 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" path="/var/lib/kubelet/pods/d215d476-bd01-4f54-b4c3-30a3e6a96c44/volumes" Nov 01 05:34:56.947891 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:34:56.947856 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:35:17.146929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.146847 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:35:17.147520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147502 4425 cpu_manager.go:414] "containerMap: removing container" podUID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" containerName="payload-processing" Nov 01 05:35:17.147568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147524 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" containerName="payload-processing" Nov 01 05:35:17.147568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147548 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" containerName="payload-pre-processing" Nov 01 05:35:17.147568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147554 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" containerName="payload-pre-processing" Nov 01 05:35:17.147675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147575 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" containerName="revoke-keys" Nov 01 05:35:17.147675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147580 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" containerName="revoke-keys" Nov 01 05:35:17.147675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147671 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fc3782-aab8-415a-ac6d-0f18e05a3ee5" containerName="payload-processing" Nov 01 05:35:17.147767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147681 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d215d476-bd01-4f54-b4c3-30a3e6a96c44" containerName="payload-pre-processing" Nov 01 05:35:17.147767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.147692 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" containerName="revoke-keys" Nov 01 05:35:17.152619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.152600 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.207849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.207817 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:35:17.255069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.255031 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"kube-root-ca.crt\"" Nov 01 05:35:17.256481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.256459 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"default-dockercfg-k2c98\"" Nov 01 05:35:17.256481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.256459 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"auth-test-model-afc555-kserve-self-signed-certs\"" Nov 01 05:35:17.256649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.256461 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"openshift-service-ca.crt\"" Nov 01 05:35:17.324214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324183 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.324423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324293 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.324423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324342 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.324533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324448 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.324590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324560 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.324637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.324601 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqg6\" (UniqueName: \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426128 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426197 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426279 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426371 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426403 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqg6\" (UniqueName: \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426498 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426634 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426712 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.426812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.426760 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.428779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.428751 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.429047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.429026 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.488205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.488173 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqg6\" (UniqueName: \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\") pod \"auth-test-model-afc555-kserve-7546699ddf-gml4m\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:17.765466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:17.765369 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:18.148720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:18.148687 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:35:18.150365 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:18.150316 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da5c379_5397_4727_ad13_1f4450693c83.slice/crio-b63b74d1340d3051faba56ebbf74016378e765079310a3e72dd429a6ff533bcd WatchSource:0}: Error finding container b63b74d1340d3051faba56ebbf74016378e765079310a3e72dd429a6ff533bcd: Status 404 returned error can't find the container with id b63b74d1340d3051faba56ebbf74016378e765079310a3e72dd429a6ff533bcd Nov 01 05:35:18.779559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:18.779517 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" event={"ID":"0da5c379-5397-4727-ad13-1f4450693c83","Type":"ContainerStarted","Data":"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9"} Nov 01 05:35:18.779749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:18.779576 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" event={"ID":"0da5c379-5397-4727-ad13-1f4450693c83","Type":"ContainerStarted","Data":"b63b74d1340d3051faba56ebbf74016378e765079310a3e72dd429a6ff533bcd"} Nov 01 05:35:18.900199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:18.900136 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" podStartSLOduration=1.900120507 podStartE2EDuration="1.900120507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:18.89966495 +0000 UTC m=+1934.040253870" watchObservedRunningTime="2026-11-01 05:35:18.900120507 +0000 UTC m=+1934.040709426" Nov 01 05:35:19.173346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.173305 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7"] Nov 01 05:35:19.176346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.176322 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec52ff90-sd5v7"] Nov 01 05:35:19.244883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.244846 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:35:19.245179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.245116 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" containerID="cri-o://383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05" gracePeriod=30 Nov 01 05:35:19.667122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.667091 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr"] Nov 01 05:35:19.671143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.671123 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.736568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.736540 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:35:19.746010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.745964 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:35:19.749094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.749070 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr"] Nov 01 05:35:19.754714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.754683 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.755012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.754972 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgs4z\" (UniqueName: \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.755130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.755033 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.765668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.765643 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:19.782483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.782434 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:35:19.782483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.782479 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:35:19.856211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.856175 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgs4z\" (UniqueName: \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.856419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.856226 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.856419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.856326 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.857049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.856972 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.858745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.858723 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.920280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.920177 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgs4z\" (UniqueName: \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\") pod \"maas-api-revoke-sub-d2cda842-4dgrr\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:19.990427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:19.990384 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:20.370572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.370539 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr"] Nov 01 05:35:20.371140 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:20.371115 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33398d25_592f_44cd_bd27_162e8c6f1ec4.slice/crio-2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5 WatchSource:0}: Error finding container 2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5: Status 404 returned error can't find the container with id 2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5 Nov 01 05:35:20.756083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.756053 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:35:20.761635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.761615 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.804614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.804575 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" event={"ID":"33398d25-592f-44cd-bd27-162e8c6f1ec4","Type":"ContainerStarted","Data":"2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5"} Nov 01 05:35:20.816793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.816760 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c6c6fdb9-gw-openshift-default-dockercfg-wrwsb\"" Nov 01 05:35:20.860997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.860968 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:35:20.871488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871452 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871516 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871565 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871604 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9xg\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871689 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871750 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871783 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.871955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871843 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.872084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.871970 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973247 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973345 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973373 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973406 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973436 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9xg\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973499 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973547 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973575 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973621 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.973957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.973797 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.974444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.974020 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.974444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.974059 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.974444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.974211 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.974444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.974384 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.975094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.975072 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb41c0ca-11e1-4c31-bdf0-b590e389cbb9" path="/var/lib/kubelet/pods/cb41c0ca-11e1-4c31-bdf0-b590e389cbb9/volumes" Nov 01 05:35:20.976468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.976442 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:20.976586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:20.976468 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:21.035889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.035852 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:21.036073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.036053 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9xg\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\") pod \"e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:21.079580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.079543 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:21.513426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.513397 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:35:21.513825 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:21.513804 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0156f43e_648a_4f87_9caa_08f54b6bfb86.slice/crio-712d15415ac55c9aff82f8f48d0631d866e4cca199ebc9c509b02a5238fbeb50 WatchSource:0}: Error finding container 712d15415ac55c9aff82f8f48d0631d866e4cca199ebc9c509b02a5238fbeb50: Status 404 returned error can't find the container with id 712d15415ac55c9aff82f8f48d0631d866e4cca199ebc9c509b02a5238fbeb50 Nov 01 05:35:21.516698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.516655 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:21.516793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.516754 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:21.516832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.516799 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:21.814617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.814581 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerStarted","Data":"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385"} Nov 01 05:35:21.814819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.814660 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerStarted","Data":"712d15415ac55c9aff82f8f48d0631d866e4cca199ebc9c509b02a5238fbeb50"} Nov 01 05:35:21.816331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.816307 4425 generic.go:356] "Generic (PLEG): container finished" podID="33398d25-592f-44cd-bd27-162e8c6f1ec4" containerID="9f471d47c35984eb8d44d73b2f137f567a069fba87ca3867e3f13fe6fb8bd168" exitCode=0 Nov 01 05:35:21.816461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.816366 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" event={"ID":"33398d25-592f-44cd-bd27-162e8c6f1ec4","Type":"ContainerDied","Data":"9f471d47c35984eb8d44d73b2f137f567a069fba87ca3867e3f13fe6fb8bd168"} Nov 01 05:35:21.938706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:21.938634 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podStartSLOduration=1.938614708 podStartE2EDuration="1.938614708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:21.936573866 +0000 UTC m=+1937.077162806" watchObservedRunningTime="2026-11-01 05:35:21.938614708 +0000 UTC m=+1937.079203628" Nov 01 05:35:22.080723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:22.080633 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:22.086686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:22.086661 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:22.823323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:22.823282 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:22.824344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:22.824325 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:35:22.983409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:22.983379 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:23.099411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.099326 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\") pod \"33398d25-592f-44cd-bd27-162e8c6f1ec4\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " Nov 01 05:35:23.099587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.099487 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\") pod \"33398d25-592f-44cd-bd27-162e8c6f1ec4\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " Nov 01 05:35:23.099587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.099527 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\" (UniqueName: \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\") pod \"33398d25-592f-44cd-bd27-162e8c6f1ec4\" (UID: \"33398d25-592f-44cd-bd27-162e8c6f1ec4\") " Nov 01 05:35:23.100082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.099773 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca" pod "33398d25-592f-44cd-bd27-162e8c6f1ec4" (UID: "33398d25-592f-44cd-bd27-162e8c6f1ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:35:23.100891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.100370 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:23.103858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.103830 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca" pod "33398d25-592f-44cd-bd27-162e8c6f1ec4" (UID: "33398d25-592f-44cd-bd27-162e8c6f1ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:35:23.104000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.103876 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z" pod "33398d25-592f-44cd-bd27-162e8c6f1ec4" (UID: "33398d25-592f-44cd-bd27-162e8c6f1ec4"). InnerVolumeSpecName "kube-api-access-zgs4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:35:23.201686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.201652 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33398d25-592f-44cd-bd27-162e8c6f1ec4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:23.201686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.201684 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgs4z\" (UniqueName: \"kubernetes.io/projected/33398d25-592f-44cd-bd27-162e8c6f1ec4-kube-api-access-zgs4z\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:23.830311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.830269 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" Nov 01 05:35:23.830311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.830293 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr" event={"ID":"33398d25-592f-44cd-bd27-162e8c6f1ec4","Type":"ContainerDied","Data":"2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5"} Nov 01 05:35:23.830872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:23.830380 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2517186526d71b615c58d91c17ddf2c645d6fd57eaf8a2612af98bc3df6240a5" Nov 01 05:35:24.524693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.524633 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:35:24.619056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619026 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619177 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619285 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619322 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619373 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619381 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:35:24.619409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619399 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619473 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619506 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619569 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\") pod \"9e325088-1cc0-437a-92d6-2b0afff19b6f\" (UID: \"9e325088-1cc0-437a-92d6-2b0afff19b6f\") " Nov 01 05:35:24.619724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619667 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:35:24.619724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:35:24.619806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619712 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:35:24.619862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.619844 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:35:24.620164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.620140 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.620458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.620168 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.620458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.620184 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e325088-1cc0-437a-92d6-2b0afff19b6f-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.620458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.620198 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.620458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.620212 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.621789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.621761 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:35:24.621974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.621946 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "kube-api-access-s5th6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:35:24.621974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.621961 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:35:24.622139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.622025 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token" pod "9e325088-1cc0-437a-92d6-2b0afff19b6f" (UID: "9e325088-1cc0-437a-92d6-2b0afff19b6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:35:24.721759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.721712 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.721759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.721755 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5th6\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-kube-api-access-s5th6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.721759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.721766 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.721759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.721777 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e325088-1cc0-437a-92d6-2b0afff19b6f-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:35:24.839509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.839470 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerID="383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05" exitCode=0 Nov 01 05:35:24.839929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.839531 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" Nov 01 05:35:24.839929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.839530 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerDied","Data":"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05"} Nov 01 05:35:24.839929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.839605 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j" event={"ID":"9e325088-1cc0-437a-92d6-2b0afff19b6f","Type":"ContainerDied","Data":"353146f152d939d80964f2e929e26d4558b358b7eda2d4d7cd3257cb61f8ce7e"} Nov 01 05:35:24.839929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.839639 4425 scope.go:122] "RemoveContainer" containerID="383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05" Nov 01 05:35:24.851440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.851416 4425 scope.go:122] "RemoveContainer" containerID="c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634" Nov 01 05:35:24.860448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.860426 4425 scope.go:122] "RemoveContainer" containerID="383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05" Nov 01 05:35:24.860756 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:35:24.860730 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05\": container with ID starting with 383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05 not found: ID does not exist" containerID="383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05" Nov 01 05:35:24.860805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.860759 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05"} err="failed to get container status \"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05\": rpc error: code = NotFound desc = could not find container \"383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05\": container with ID starting with 383db6ebd910998ec46825aed1ecd0262ff88ef870b760eb0ad30d83257e8e05 not found: ID does not exist" Nov 01 05:35:24.860805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.860776 4425 scope.go:122] "RemoveContainer" containerID="c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634" Nov 01 05:35:24.861008 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:35:24.860991 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634\": container with ID starting with c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634 not found: ID does not exist" containerID="c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634" Nov 01 05:35:24.861048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:24.861014 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634"} err="failed to get container status \"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634\": rpc error: code = NotFound desc = could not find container \"c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634\": container with ID starting with c5b6b7a5e2bd0c407710036966578dfc7bcfbd28b4e391b099eec0c26ef6e634 not found: ID does not exist" Nov 01 05:35:25.011032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:25.010997 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:35:25.015798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:25.015772 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ec52ff90-openshift-default-67674f5765-7k75j"] Nov 01 05:35:26.949430 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:35:26.949395 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:35:26.973617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:26.973589 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" path="/var/lib/kubelet/pods/9e325088-1cc0-437a-92d6-2b0afff19b6f/volumes" Nov 01 05:35:27.382523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.382489 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:35:27.383094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383081 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383096 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383121 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383126 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383144 4425 cpu_manager.go:414] "containerMap: removing container" podUID="33398d25-592f-44cd-bd27-162e8c6f1ec4" containerName="revoke-keys" Nov 01 05:35:27.383231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383149 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33398d25-592f-44cd-bd27-162e8c6f1ec4" containerName="revoke-keys" Nov 01 05:35:27.383312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383231 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383242 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e325088-1cc0-437a-92d6-2b0afff19b6f" containerName="istio-proxy" Nov 01 05:35:27.383312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.383267 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="33398d25-592f-44cd-bd27-162e8c6f1ec4" containerName="revoke-keys" Nov 01 05:35:27.388117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.388101 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.444288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.444242 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:35:27.490935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.490886 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c6c6fdb9-dockercfg-7xcxz\"" Nov 01 05:35:27.491462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.491420 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c6c6fdb9\"" Nov 01 05:35:27.544697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.544670 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:35:27.548986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.548965 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.553262 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.553231 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.553453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.553432 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj8r\" (UniqueName: \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.593161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.593113 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:35:27.654648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.654556 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.654822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.654675 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.654822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.654707 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbjv\" (UniqueName: \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.654921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.654834 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj8r\" (UniqueName: \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.655371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.655348 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.717141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.717091 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj8r\" (UniqueName: \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\") pod \"payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:27.755897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.755864 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.756067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.755905 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbjv\" (UniqueName: \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.756620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.756601 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.765991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.765958 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:27.778866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.778842 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:27.818525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.818494 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbjv\" (UniqueName: \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\") pod \"payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.863946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.863906 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:27.871222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:27.871202 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:35:28.001645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.001611 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:28.244811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.244777 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:35:28.245242 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:28.245221 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92e707b_1f6a_4f43_ae91_bfd1a4e3dc5d.slice/crio-f8b2f2fad9a9020caacdff55c3a53ced20f53d7c5b68c4b3267ffe10c2ae1912 WatchSource:0}: Error finding container f8b2f2fad9a9020caacdff55c3a53ced20f53d7c5b68c4b3267ffe10c2ae1912: Status 404 returned error can't find the container with id f8b2f2fad9a9020caacdff55c3a53ced20f53d7c5b68c4b3267ffe10c2ae1912 Nov 01 05:35:28.388175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.388150 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:35:28.388612 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:28.388589 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11592a8_22a8_44a6_8e05_b04409831f68.slice/crio-446e8ad99ffb7f30978e0d6878ff28163d9c4a5fb3cda95e218fe027e186940d WatchSource:0}: Error finding container 446e8ad99ffb7f30978e0d6878ff28163d9c4a5fb3cda95e218fe027e186940d: Status 404 returned error can't find the container with id 446e8ad99ffb7f30978e0d6878ff28163d9c4a5fb3cda95e218fe027e186940d Nov 01 05:35:28.868531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.868488 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" event={"ID":"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d","Type":"ContainerStarted","Data":"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a"} Nov 01 05:35:28.868750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.868559 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" event={"ID":"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d","Type":"ContainerStarted","Data":"f8b2f2fad9a9020caacdff55c3a53ced20f53d7c5b68c4b3267ffe10c2ae1912"} Nov 01 05:35:28.868750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.868730 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:28.870401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.870373 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" event={"ID":"d11592a8-22a8-44a6-8e05-b04409831f68","Type":"ContainerStarted","Data":"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b"} Nov 01 05:35:28.870545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.870418 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" event={"ID":"d11592a8-22a8-44a6-8e05-b04409831f68","Type":"ContainerStarted","Data":"446e8ad99ffb7f30978e0d6878ff28163d9c4a5fb3cda95e218fe027e186940d"} Nov 01 05:35:28.870625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.870581 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:28.987196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:28.987120 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" podStartSLOduration=1.987100563 podStartE2EDuration="1.987100563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:28.985885782 +0000 UTC m=+1944.126474715" watchObservedRunningTime="2026-11-01 05:35:28.987100563 +0000 UTC m=+1944.127689484" Nov 01 05:35:29.106273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:29.106195 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" podStartSLOduration=2.106176931 podStartE2EDuration="2.106176931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:29.103412475 +0000 UTC m=+1944.244001471" watchObservedRunningTime="2026-11-01 05:35:29.106176931 +0000 UTC m=+1944.246765850" Nov 01 05:35:31.102366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:31.102329 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2"] Nov 01 05:35:31.151143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:31.151104 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076880c2-nfjl2"] Nov 01 05:35:32.973539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:32.973506 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a70d4a-6741-4247-b09a-a5e49c21a653" path="/var/lib/kubelet/pods/40a70d4a-6741-4247-b09a-a5e49c21a653/volumes" Nov 01 05:35:39.879546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:39.879517 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:35:39.879933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:39.879588 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:35:45.048326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:45.048291 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc"] Nov 01 05:35:45.098982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:45.098948 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eec49167-pzjcc"] Nov 01 05:35:46.974932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:46.974897 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8beec8-a107-4835-be2f-a96cbcb30ed7" path="/var/lib/kubelet/pods/ae8beec8-a107-4835-be2f-a96cbcb30ed7/volumes" Nov 01 05:35:50.882379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.882345 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:35:50.887800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.887780 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:50.935819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.935779 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:35:50.941062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.941033 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:50.949485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.949455 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\"" Nov 01 05:35:50.989061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.989031 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:35:50.998150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:50.998121 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:35:51.039955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.039918 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.039984 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hngg\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040106 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040144 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040162 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040324 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040354 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040392 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040495 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040530 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvpw\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040548 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040570 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040654 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040691 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040742 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.040818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040772 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040818 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.040924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.040905 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.141752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141658 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.141752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141702 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.141752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141722 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.141752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141752 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141771 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141797 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141827 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141859 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvpw\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141890 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141920 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141952 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.141987 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142017 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142051 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.142090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142077 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142125 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142167 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142208 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142281 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hngg\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142294 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142360 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142478 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142618 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142886 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.142207 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.143539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.143396 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.143387 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.143744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.143723 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.144513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.144493 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.145240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.145215 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.145518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.145492 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.145567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.145529 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.204955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.204924 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.205109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.205010 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.205174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.205113 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvpw\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\") pod \"maas-default-gateway-openshift-default-58d898bd55-tmwv2\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.205385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.205361 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hngg\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\") pod \"maas-default-gateway-openshift-default-58d898bd55-gxmrv\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.262070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.262039 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:51.504127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.503030 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:51.724622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.724586 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:35:51.724848 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:51.724823 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb24c12_7422_4b1d_9977_214836baf59a.slice/crio-efa4a7f8364892dc1db17c7007bd463ca86ccce5b85d90113d081383e8da0b9d WatchSource:0}: Error finding container efa4a7f8364892dc1db17c7007bd463ca86ccce5b85d90113d081383e8da0b9d: Status 404 returned error can't find the container with id efa4a7f8364892dc1db17c7007bd463ca86ccce5b85d90113d081383e8da0b9d Nov 01 05:35:51.727538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.727505 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:51.727621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.727570 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:51.727621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.727599 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:51.912985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.912953 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:35:51.913588 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:51.913557 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c9f4f7_d918_4d97_8e43_ac63af55081e.slice/crio-9e79aa7943c707a440143c735f428107ec8a5bc3daef80018ef836d24ae380ea WatchSource:0}: Error finding container 9e79aa7943c707a440143c735f428107ec8a5bc3daef80018ef836d24ae380ea: Status 404 returned error can't find the container with id 9e79aa7943c707a440143c735f428107ec8a5bc3daef80018ef836d24ae380ea Nov 01 05:35:51.916127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.916091 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:51.916194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.916170 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:51.916232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:51.916207 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:35:52.022988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.022901 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" event={"ID":"49c9f4f7-d918-4d97-8e43-ac63af55081e","Type":"ContainerStarted","Data":"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f"} Nov 01 05:35:52.022988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.022965 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" event={"ID":"49c9f4f7-d918-4d97-8e43-ac63af55081e","Type":"ContainerStarted","Data":"9e79aa7943c707a440143c735f428107ec8a5bc3daef80018ef836d24ae380ea"} Nov 01 05:35:52.025576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.025545 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" event={"ID":"5eb24c12-7422-4b1d-9977-214836baf59a","Type":"ContainerStarted","Data":"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816"} Nov 01 05:35:52.025711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.025599 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" event={"ID":"5eb24c12-7422-4b1d-9977-214836baf59a","Type":"ContainerStarted","Data":"efa4a7f8364892dc1db17c7007bd463ca86ccce5b85d90113d081383e8da0b9d"} Nov 01 05:35:52.145849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.145785 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podStartSLOduration=2.145765072 podStartE2EDuration="2.145765072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:52.144902581 +0000 UTC m=+1967.285491501" watchObservedRunningTime="2026-11-01 05:35:52.145765072 +0000 UTC m=+1967.286354181" Nov 01 05:35:52.263004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.262956 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:35:52.264905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.264846 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:52.264905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.264903 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:52.267466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.267412 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podStartSLOduration=2.267393994 podStartE2EDuration="2.267393994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:52.267283841 +0000 UTC m=+1967.407872786" watchObservedRunningTime="2026-11-01 05:35:52.267393994 +0000 UTC m=+1967.407982914" Nov 01 05:35:52.505279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.503588 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:35:52.505965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.505932 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:52.506191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:52.506164 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:53.273278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:53.262676 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:53.273278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:53.262740 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:53.504420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:53.503840 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:53.504420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:53.503901 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:54.271768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:54.271720 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:54.272080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:54.272050 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:54.504234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:54.504191 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:54.504738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:54.504280 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:55.264705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:55.264666 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:55.265009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:55.264977 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:55.504209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:55.504172 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:55.504432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:55.504239 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:56.263281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.262686 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:56.263281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.262747 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:56.492350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.492307 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:35:56.498203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.498177 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.503882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.503854 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:56.504021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.503901 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:56.520041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.519978 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:35:56.532072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.532040 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.532280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.532233 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.532361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.532322 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.532407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.532366 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8qz\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.571290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.566411 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-500efa28b76e\"" Nov 01 05:35:56.633228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.633191 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.633228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.633230 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.633537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.633276 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8qz\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.633537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.633380 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.633918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.633890 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.634127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.634106 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.636359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.636340 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.697146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.697100 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8qz\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:56.813736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:56.813708 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:57.009045 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:35:57.009008 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:35:57.097528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.097390 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn"] Nov 01 05:35:57.103078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.103048 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb58d33-zccmn"] Nov 01 05:35:57.280300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.275680 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:57.280300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.275745 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:57.280300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.277884 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:35:57.294466 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:35:57.294150 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f2c171_434e_4ebc_a6bd_43eb54eef91c.slice/crio-4b2db9dd20d3b68c86f868c7523dba04bc59388199e49c6d4162e289f0d81b93 WatchSource:0}: Error finding container 4b2db9dd20d3b68c86f868c7523dba04bc59388199e49c6d4162e289f0d81b93: Status 404 returned error can't find the container with id 4b2db9dd20d3b68c86f868c7523dba04bc59388199e49c6d4162e289f0d81b93 Nov 01 05:35:57.503738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.503703 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:57.503893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:57.503765 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:58.113329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.113285 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" event={"ID":"e2f2c171-434e-4ebc-a6bd-43eb54eef91c","Type":"ContainerStarted","Data":"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf"} Nov 01 05:35:58.113747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.113383 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:35:58.113747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.113403 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" event={"ID":"e2f2c171-434e-4ebc-a6bd-43eb54eef91c","Type":"ContainerStarted","Data":"4b2db9dd20d3b68c86f868c7523dba04bc59388199e49c6d4162e289f0d81b93"} Nov 01 05:35:58.233355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.233285 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" podStartSLOduration=2.233246612 podStartE2EDuration="2.233246612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:58.232672746 +0000 UTC m=+1973.373261689" watchObservedRunningTime="2026-11-01 05:35:58.233246612 +0000 UTC m=+1973.373835532" Nov 01 05:35:58.262543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.262505 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:58.262744 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.262573 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:58.504708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.504619 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:58.504708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.504688 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:35:58.975776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:58.975737 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334fd691-5f20-411d-9de9-45a504a8f5c4" path="/var/lib/kubelet/pods/334fd691-5f20-411d-9de9-45a504a8f5c4/volumes" Nov 01 05:35:59.262601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:59.262512 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 01 05:35:59.262977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:59.262593 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 01 05:35:59.504693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:59.504655 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-gxmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 01 05:35:59.504873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:35:59.504732 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 01 05:36:00.397446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.397409 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:36:00.399060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.399030 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:36:00.399982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.399948 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:36:00.509666 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.509637 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:36:00.509963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.509937 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:36:00.510667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:00.510648 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:36:06.293049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:06.293013 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:36:09.128274 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:09.128227 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:36:09.316736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:09.316670 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:36:11.294161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.294113 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" podUID="a71eac8a-d6c1-4080-8244-5626c5b65195" containerName="payload-processing" containerID="cri-o://ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32" gracePeriod=25 Nov 01 05:36:11.594223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.594182 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:36:11.713382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.712227 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\") pod \"a71eac8a-d6c1-4080-8244-5626c5b65195\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " Nov 01 05:36:11.713382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.712333 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\") pod \"a71eac8a-d6c1-4080-8244-5626c5b65195\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " Nov 01 05:36:11.713382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.712488 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\") pod \"a71eac8a-d6c1-4080-8244-5626c5b65195\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " Nov 01 05:36:11.713382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.712576 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\") pod \"a71eac8a-d6c1-4080-8244-5626c5b65195\" (UID: \"a71eac8a-d6c1-4080-8244-5626c5b65195\") " Nov 01 05:36:11.713382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.713059 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume" pod "a71eac8a-d6c1-4080-8244-5626c5b65195" (UID: "a71eac8a-d6c1-4080-8244-5626c5b65195"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:11.714083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.713702 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume" pod "a71eac8a-d6c1-4080-8244-5626c5b65195" (UID: "a71eac8a-d6c1-4080-8244-5626c5b65195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:11.716692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.716655 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials" pod "a71eac8a-d6c1-4080-8244-5626c5b65195" (UID: "a71eac8a-d6c1-4080-8244-5626c5b65195"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:11.717800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.717775 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd" pod "a71eac8a-d6c1-4080-8244-5626c5b65195" (UID: "a71eac8a-d6c1-4080-8244-5626c5b65195"). InnerVolumeSpecName "kube-api-access-khmrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:11.814676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.814594 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmrd\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-kube-api-access-khmrd\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:11.814676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.814624 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a71eac8a-d6c1-4080-8244-5626c5b65195-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:11.814676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.814635 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:11.814676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:11.814665 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71eac8a-d6c1-4080-8244-5626c5b65195-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:12.225022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.224928 4425 generic.go:356] "Generic (PLEG): container finished" podID="a71eac8a-d6c1-4080-8244-5626c5b65195" containerID="ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32" exitCode=0 Nov 01 05:36:12.225022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.224980 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" Nov 01 05:36:12.225242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.225020 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" event={"ID":"a71eac8a-d6c1-4080-8244-5626c5b65195","Type":"ContainerDied","Data":"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32"} Nov 01 05:36:12.225242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.225081 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb" event={"ID":"a71eac8a-d6c1-4080-8244-5626c5b65195","Type":"ContainerDied","Data":"f992d7be350ff3bdee04f344ec3a6a8f7452f169f5f5805429cf615f043e9134"} Nov 01 05:36:12.225242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.225110 4425 scope.go:122] "RemoveContainer" containerID="ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32" Nov 01 05:36:12.236354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.236329 4425 scope.go:122] "RemoveContainer" containerID="ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32" Nov 01 05:36:12.236648 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:36:12.236628 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32\": container with ID starting with ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32 not found: ID does not exist" containerID="ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32" Nov 01 05:36:12.236706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.236656 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32"} err="failed to get container status \"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32\": rpc error: code = NotFound desc = could not find container \"ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32\": container with ID starting with ba39abf68e6900b009a9a3458d57d4308cdb07d4b369aa5ed7b5a02c42c05a32 not found: ID does not exist" Nov 01 05:36:12.393523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.393486 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:36:12.397435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.397409 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc4twb"] Nov 01 05:36:12.974035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:12.973998 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71eac8a-d6c1-4080-8244-5626c5b65195" path="/var/lib/kubelet/pods/a71eac8a-d6c1-4080-8244-5626c5b65195/volumes" Nov 01 05:36:14.049515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.049482 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4"] Nov 01 05:36:14.052476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.052454 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6570b548-2kfn4"] Nov 01 05:36:14.317822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.317776 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" podUID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" containerName="payload-processing" containerID="cri-o://62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8" gracePeriod=25 Nov 01 05:36:14.590939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.590914 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:36:14.663649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.663611 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\") pod \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " Nov 01 05:36:14.663823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.663737 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\") pod \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " Nov 01 05:36:14.663901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.663883 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\") pod \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " Nov 01 05:36:14.663973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.663956 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\") pod \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\" (UID: \"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8\") " Nov 01 05:36:14.664151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.664123 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume" pod "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" (UID: "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:14.664476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.664391 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume" pod "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" (UID: "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:14.664544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.664502 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:14.664544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.664519 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:14.666045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.666027 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl" pod "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" (UID: "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8"). InnerVolumeSpecName "kube-api-access-65knl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:14.666146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.666119 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials" pod "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" (UID: "fe4c162b-05fb-4f3f-9c3d-6087be2f56c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:14.765602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.765567 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:14.765602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.765601 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65knl\" (UniqueName: \"kubernetes.io/projected/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8-kube-api-access-65knl\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:14.974486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.974400 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf3345e-6458-46f3-b78b-9c4734f38fb2" path="/var/lib/kubelet/pods/edf3345e-6458-46f3-b78b-9c4734f38fb2/volumes" Nov 01 05:36:14.999446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:14.999402 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:36:15.000426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000406 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a71eac8a-d6c1-4080-8244-5626c5b65195" containerName="payload-processing" Nov 01 05:36:15.000476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000431 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71eac8a-d6c1-4080-8244-5626c5b65195" containerName="payload-processing" Nov 01 05:36:15.000476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000454 4425 cpu_manager.go:414] "containerMap: removing container" podUID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" containerName="payload-processing" Nov 01 05:36:15.000538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000476 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" containerName="payload-processing" Nov 01 05:36:15.000653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000639 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71eac8a-d6c1-4080-8244-5626c5b65195" containerName="payload-processing" Nov 01 05:36:15.000688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.000660 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" containerName="payload-processing" Nov 01 05:36:15.004944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.004921 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.060244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.060217 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"kube-root-ca.crt\"" Nov 01 05:36:15.064779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.064755 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"openshift-service-ca.crt\"" Nov 01 05:36:15.068864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.068837 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.068954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.068879 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmz2\" (UniqueName: \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.068954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.068900 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.069102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.069082 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.069225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.069212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.069285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.069273 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.113700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.112990 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"auth-test-model-1c4191-kserve-self-signed-certs\"" Nov 01 05:36:15.113700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.113182 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"default-dockercfg-nqjr7\"" Nov 01 05:36:15.123222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.123186 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:36:15.170610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.170821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170620 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmz2\" (UniqueName: \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.170821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170642 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.170821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.170821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170802 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.171064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.170845 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.171127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.171043 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.171213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.171186 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.171306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.171182 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.173504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.173480 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.173901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.173882 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.235740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.235671 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmz2\" (UniqueName: \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\") pod \"auth-test-model-1c4191-kserve-76fff7877c-2w5q9\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.255522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.255484 4425 generic.go:356] "Generic (PLEG): container finished" podID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" containerID="62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8" exitCode=0 Nov 01 05:36:15.255676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.255537 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" Nov 01 05:36:15.255676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.255562 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" event={"ID":"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8","Type":"ContainerDied","Data":"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8"} Nov 01 05:36:15.255676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.255645 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn" event={"ID":"fe4c162b-05fb-4f3f-9c3d-6087be2f56c8","Type":"ContainerDied","Data":"ce8bc6147bdff4beb17561f6ac40d74987997475bc26eb5f0b456ee63b0be32b"} Nov 01 05:36:15.255676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.255670 4425 scope.go:122] "RemoveContainer" containerID="62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8" Nov 01 05:36:15.265320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.265301 4425 scope.go:122] "RemoveContainer" containerID="62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8" Nov 01 05:36:15.265602 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:36:15.265584 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8\": container with ID starting with 62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8 not found: ID does not exist" containerID="62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8" Nov 01 05:36:15.265663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.265609 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8"} err="failed to get container status \"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8\": rpc error: code = NotFound desc = could not find container \"62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8\": container with ID starting with 62f60683e7ba9e7e1a00ca0cc368ecbc87df7785cf6a19ba9fb52bb2633d71a8 not found: ID does not exist" Nov 01 05:36:15.319470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.319438 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:15.436529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.435567 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:36:15.448469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.448437 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c67qfjzn"] Nov 01 05:36:15.702510 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:36:15.702478 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4770db10_c48e_4ac9_adb1_f2ed00d5e4e4.slice/crio-d9033505552ce3416206bf9b0e947c69e6cc528cfe14bb56e2c384e63143d2ef WatchSource:0}: Error finding container d9033505552ce3416206bf9b0e947c69e6cc528cfe14bb56e2c384e63143d2ef: Status 404 returned error can't find the container with id d9033505552ce3416206bf9b0e947c69e6cc528cfe14bb56e2c384e63143d2ef Nov 01 05:36:15.702920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:15.702900 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:36:16.264594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:16.264552 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" event={"ID":"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4","Type":"ContainerStarted","Data":"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583"} Nov 01 05:36:16.265026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:16.264601 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" event={"ID":"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4","Type":"ContainerStarted","Data":"d9033505552ce3416206bf9b0e947c69e6cc528cfe14bb56e2c384e63143d2ef"} Nov 01 05:36:16.319811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:16.319779 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:16.383955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:16.383896 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" podStartSLOduration=2.383878388 podStartE2EDuration="2.383878388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:16.382637449 +0000 UTC m=+1991.523226385" watchObservedRunningTime="2026-11-01 05:36:16.383878388 +0000 UTC m=+1991.524467308" Nov 01 05:36:16.974106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:16.974068 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4c162b-05fb-4f3f-9c3d-6087be2f56c8" path="/var/lib/kubelet/pods/fe4c162b-05fb-4f3f-9c3d-6087be2f56c8/volumes" Nov 01 05:36:20.048163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:20.048126 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq"] Nov 01 05:36:20.102187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:20.102144 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006efc36-pv5wq"] Nov 01 05:36:20.974076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:20.974037 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77db025-c009-4cbe-9f41-64737b3b0ac6" path="/var/lib/kubelet/pods/c77db025-c009-4cbe-9f41-64737b3b0ac6/volumes" Nov 01 05:36:21.879790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.879758 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:36:21.885538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.885514 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:21.938773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.938738 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-388566305a92\"" Nov 01 05:36:21.955502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.955472 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:21.955660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.955528 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:21.955760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.955738 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:21.955827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.955787 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpws\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:21.992910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:21.992876 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:36:22.056885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.056847 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.057191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.056896 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.057191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.056948 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.057191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.056972 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpws\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.057665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.057637 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.057836 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.057815 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.059919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.059901 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.119016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.118981 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpws\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.199782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.199700 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:22.579067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:22.579040 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:36:22.580708 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:36:22.580682 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f496e24_030a_43ca_a04c_bf00a788c3d6.slice/crio-865784bea716737206e0128b81d3ddab0f961f69be39a42a9d01adf38fe7cd13 WatchSource:0}: Error finding container 865784bea716737206e0128b81d3ddab0f961f69be39a42a9d01adf38fe7cd13: Status 404 returned error can't find the container with id 865784bea716737206e0128b81d3ddab0f961f69be39a42a9d01adf38fe7cd13 Nov 01 05:36:23.312880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:23.312844 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" event={"ID":"7f496e24-030a-43ca-a04c-bf00a788c3d6","Type":"ContainerStarted","Data":"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3"} Nov 01 05:36:23.313322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:23.313011 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:23.313322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:23.313050 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" event={"ID":"7f496e24-030a-43ca-a04c-bf00a788c3d6","Type":"ContainerStarted","Data":"865784bea716737206e0128b81d3ddab0f961f69be39a42a9d01adf38fe7cd13"} Nov 01 05:36:23.432474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:23.432413 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" podStartSLOduration=2.4323966 podStartE2EDuration="2.4323966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:23.431336821 +0000 UTC m=+1998.571925740" watchObservedRunningTime="2026-11-01 05:36:23.4323966 +0000 UTC m=+1998.572985510" Nov 01 05:36:24.491629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.491597 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:36:24.496212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.496192 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.550389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.550351 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\"" Nov 01 05:36:24.589305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.589224 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.589541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.589518 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.589643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.589618 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.589708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.589664 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5ww\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.600773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600743 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\"" Nov 01 05:36:24.600773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600752 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\"" Nov 01 05:36:24.600977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600834 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\"" Nov 01 05:36:24.600977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600857 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-31f7d618cb24\"" Nov 01 05:36:24.600977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600893 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-3e39090bc554\"" Nov 01 05:36:24.600977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.600914 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-8eb61444db22\"" Nov 01 05:36:24.601192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.601009 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\"" Nov 01 05:36:24.601192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.601062 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-e2e--8ea5e171-dockercfg-hr8xq\"" Nov 01 05:36:24.604342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.604324 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:36:24.690458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.690423 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.690636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.690478 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.690636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.690506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5ww\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.690636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.690576 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.691179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.691156 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.691467 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.691415 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.693477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.693455 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.753359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.753272 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5ww\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:24.811061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:24.811025 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:25.195646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.195621 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:36:25.196748 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:36:25.196726 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338459c8_67a4_4e10_8a9b_8c5fa963d39e.slice/crio-490d64d460ea2ed1b14ce6ec9e220fd43534b1bbeec5f6f84c54722045a6cc13 WatchSource:0}: Error finding container 490d64d460ea2ed1b14ce6ec9e220fd43534b1bbeec5f6f84c54722045a6cc13: Status 404 returned error can't find the container with id 490d64d460ea2ed1b14ce6ec9e220fd43534b1bbeec5f6f84c54722045a6cc13 Nov 01 05:36:25.319639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.319614 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:25.331500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.331466 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" event={"ID":"338459c8-67a4-4e10-8a9b-8c5fa963d39e","Type":"ContainerStarted","Data":"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce"} Nov 01 05:36:25.331697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.331527 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" event={"ID":"338459c8-67a4-4e10-8a9b-8c5fa963d39e","Type":"ContainerStarted","Data":"490d64d460ea2ed1b14ce6ec9e220fd43534b1bbeec5f6f84c54722045a6cc13"} Nov 01 05:36:25.331883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.331578 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:25.335740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.335714 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:25.347859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.347832 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:36:25.452042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:25.451922 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" podStartSLOduration=1.451907652 podStartE2EDuration="1.451907652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:25.449964058 +0000 UTC m=+2000.590552978" watchObservedRunningTime="2026-11-01 05:36:25.451907652 +0000 UTC m=+2000.592496567" Nov 01 05:36:26.948929 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:36:26.948880 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:36:31.066282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:31.065934 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8"] Nov 01 05:36:31.108557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:31.108528 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1232cfdc-gdsl8"] Nov 01 05:36:32.974446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:32.974411 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949bbcec-3a69-4f85-8c43-e6023903b0e9" path="/var/lib/kubelet/pods/949bbcec-3a69-4f85-8c43-e6023903b0e9/volumes" Nov 01 05:36:34.323727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:34.323695 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:36:34.481881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:34.481828 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:36:36.251888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.251854 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:36:36.257415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.257397 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.302438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.302406 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:36:36.306474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.306450 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.323782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.323754 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.323914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.323810 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wzw\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.323914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.323883 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.323995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.323935 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.324030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.324017 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.324061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.324045 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.324234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.324213 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.324293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.324277 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.324356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.324341 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.342799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.342774 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:36:36.398367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.398327 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:36:36.426102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426062 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.426306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426117 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.426306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426235 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.426501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426471 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.426657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426634 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.426849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426829 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.427003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.426986 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427141 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.427314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427297 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427437 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.427609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427590 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wzw\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427730 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427674 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427732 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427854 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglpq\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.427969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427894 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.427969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427920 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.427969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427960 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.428196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.428004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.428196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.428047 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.428407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.428328 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.428523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.428498 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.428594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.427395 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.431355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.431321 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.433759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.433735 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.480062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.479293 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:36:36.491482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.491452 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wzw\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.498795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.498761 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zzhvc\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.529864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.529766 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.530038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.529879 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.530353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530307 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.530450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530424 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530514 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglpq\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530559 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530624 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530705 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530731 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530766 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530929 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530946 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.530729 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.531597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.531536 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.534266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.534200 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.535545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.535521 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.571944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.571685 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:36.593908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.593878 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.594354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.594332 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglpq\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\") pod \"maas-default-gateway-openshift-default-58d898bd55-kdh6t\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:36.620615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:36.620577 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:37.016073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.016034 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:36:37.017606 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:36:37.017577 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd74491_1ceb_4ef4_b5ef_a0a226cd2533.slice/crio-3f1f3e6796443d34f79ea6a39d422a76529ba533083ed57a25790a64d3f6bcac WatchSource:0}: Error finding container 3f1f3e6796443d34f79ea6a39d422a76529ba533083ed57a25790a64d3f6bcac: Status 404 returned error can't find the container with id 3f1f3e6796443d34f79ea6a39d422a76529ba533083ed57a25790a64d3f6bcac Nov 01 05:36:37.020339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.020303 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.020438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.020379 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.020438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.020408 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.069362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.069330 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:36:37.070793 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:36:37.070763 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254bc61_bbde_4309_b9ae_57c7859bee94.slice/crio-6f06941393938dce2c027ec607b068737001cb0771f87215395ed15ddb03da96 WatchSource:0}: Error finding container 6f06941393938dce2c027ec607b068737001cb0771f87215395ed15ddb03da96: Status 404 returned error can't find the container with id 6f06941393938dce2c027ec607b068737001cb0771f87215395ed15ddb03da96 Nov 01 05:36:37.074281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.074217 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.074383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.074321 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.074383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.074357 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.434276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.434159 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" event={"ID":"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533","Type":"ContainerStarted","Data":"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b"} Nov 01 05:36:37.434800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.434280 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" event={"ID":"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533","Type":"ContainerStarted","Data":"3f1f3e6796443d34f79ea6a39d422a76529ba533083ed57a25790a64d3f6bcac"} Nov 01 05:36:37.436483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.436455 4425 generic.go:356] "Generic (PLEG): container finished" podID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerID="b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385" exitCode=0 Nov 01 05:36:37.436610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.436536 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerDied","Data":"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385"} Nov 01 05:36:37.437013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.436993 4425 scope.go:122] "RemoveContainer" containerID="b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385" Nov 01 05:36:37.438739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.438706 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.438841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.438804 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.438908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.438857 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:36:37.439682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.439658 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" event={"ID":"9254bc61-bbde-4309-b9ae-57c7859bee94","Type":"ContainerStarted","Data":"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513"} Nov 01 05:36:37.439795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.439703 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" event={"ID":"9254bc61-bbde-4309-b9ae-57c7859bee94","Type":"ContainerStarted","Data":"6f06941393938dce2c027ec607b068737001cb0771f87215395ed15ddb03da96"} Nov 01 05:36:37.560757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.560694 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podStartSLOduration=1.5606739269999998 podStartE2EDuration="1.560673927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:37.555817029 +0000 UTC m=+2012.696405964" watchObservedRunningTime="2026-11-01 05:36:37.560673927 +0000 UTC m=+2012.701262848" Nov 01 05:36:37.572572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.572531 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:37.621683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.621643 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:37.827409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.827321 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:36:37.828982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:37.828925 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podStartSLOduration=1.828911105 podStartE2EDuration="1.828911105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:37.828011075 +0000 UTC m=+2012.968599995" watchObservedRunningTime="2026-11-01 05:36:37.828911105 +0000 UTC m=+2012.969500023" Nov 01 05:36:38.505340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:38.505303 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerStarted","Data":"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a"} Nov 01 05:36:38.589425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:38.589388 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:38.589767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:38.589730 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:38.623341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:38.623306 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:38.623876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:38.623840 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:39.087279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.086652 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:36:39.091276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.090362 4425 patch_prober.go:29] interesting pod/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 05:36:39.091276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.090418 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 05:36:39.495339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.495037 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" podUID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" containerName="payload-processing" containerID="cri-o://7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf" gracePeriod=25 Nov 01 05:36:39.574748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.574715 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:39.575388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.575343 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:39.621775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.621662 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:39.621775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:39.621718 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:40.083342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.082729 4425 patch_prober.go:29] interesting pod/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 05:36:40.083342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.082787 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 05:36:40.129874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.129802 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:36:40.229401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.229374 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\") pod \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " Nov 01 05:36:40.229574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.229531 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\") pod \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " Nov 01 05:36:40.229574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.229571 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\") pod \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " Nov 01 05:36:40.229650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.229605 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\") pod \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\" (UID: \"e2f2c171-434e-4ebc-a6bd-43eb54eef91c\") " Nov 01 05:36:40.230030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.229993 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume" pod "e2f2c171-434e-4ebc-a6bd-43eb54eef91c" (UID: "e2f2c171-434e-4ebc-a6bd-43eb54eef91c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:40.230171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.230136 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume" pod "e2f2c171-434e-4ebc-a6bd-43eb54eef91c" (UID: "e2f2c171-434e-4ebc-a6bd-43eb54eef91c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:36:40.231913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.231883 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz" pod "e2f2c171-434e-4ebc-a6bd-43eb54eef91c" (UID: "e2f2c171-434e-4ebc-a6bd-43eb54eef91c"). InnerVolumeSpecName "kube-api-access-gn8qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:40.232214 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.232194 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials" pod "e2f2c171-434e-4ebc-a6bd-43eb54eef91c" (UID: "e2f2c171-434e-4ebc-a6bd-43eb54eef91c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:36:40.331232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.330974 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:40.331232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.331005 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8qz\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-kube-api-access-gn8qz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:40.331232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.331019 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:40.331232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.331032 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2f2c171-434e-4ebc-a6bd-43eb54eef91c-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:36:40.533016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.532934 4425 generic.go:356] "Generic (PLEG): container finished" podID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" containerID="7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf" exitCode=0 Nov 01 05:36:40.533016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.532990 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" Nov 01 05:36:40.533229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.533030 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" event={"ID":"e2f2c171-434e-4ebc-a6bd-43eb54eef91c","Type":"ContainerDied","Data":"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf"} Nov 01 05:36:40.533229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.533130 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7" event={"ID":"e2f2c171-434e-4ebc-a6bd-43eb54eef91c","Type":"ContainerDied","Data":"4b2db9dd20d3b68c86f868c7523dba04bc59388199e49c6d4162e289f0d81b93"} Nov 01 05:36:40.533229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.533161 4425 scope.go:122] "RemoveContainer" containerID="7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf" Nov 01 05:36:40.575279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.572124 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:40.575279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.572201 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:40.600329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.598570 4425 scope.go:122] "RemoveContainer" containerID="7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf" Nov 01 05:36:40.619499 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:36:40.619455 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf\": container with ID starting with 7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf not found: ID does not exist" containerID="7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf" Nov 01 05:36:40.619778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.619744 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf"} err="failed to get container status \"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf\": rpc error: code = NotFound desc = could not find container \"7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf\": container with ID starting with 7758da43a918544518697d943207f80664346ff08bc08b447728787608dcb3bf not found: ID does not exist" Nov 01 05:36:40.621748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.621711 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:40.621879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.621773 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:40.731293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.730720 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:36:40.733973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:40.733946 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577b8ttg7"] Nov 01 05:36:41.020352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.020315 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" path="/var/lib/kubelet/pods/e2f2c171-434e-4ebc-a6bd-43eb54eef91c/volumes" Nov 01 05:36:41.084710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.084670 4425 patch_prober.go:29] interesting pod/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 05:36:41.085006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.084972 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 05:36:41.577153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.576662 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:41.577153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.576726 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:41.621789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.621662 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:41.621789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:41.621725 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:42.086708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.086673 4425 patch_prober.go:29] interesting pod/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 05:36:42.087034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.087004 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 05:36:42.588280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.587671 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:42.588280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.587734 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:42.621844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.621792 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:42.622053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:42.621858 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:43.105279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.104243 4425 patch_prober.go:29] interesting pod/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 01 05:36:43.105279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.104359 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 01 05:36:43.573771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.573728 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:43.573988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.573814 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:43.621424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.621385 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:43.621836 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:43.621455 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:44.085150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.085096 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:36:44.085577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.085547 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:36:44.086299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.086282 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:36:44.572451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.572414 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Nov 01 05:36:44.572617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.572499 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Nov 01 05:36:44.621738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.621700 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 05:36:44.622398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:44.622365 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 05:36:45.947071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.947039 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:45.948303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.948245 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:45.950447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.950427 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:36:45.985162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.985132 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:45.985699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.985578 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:45.986988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:36:45.986968 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:36:56.951357 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:36:56.951279 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:37:01.302130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.302050 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb"] Nov 01 05:37:01.302693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.302678 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" containerName="payload-processing" Nov 01 05:37:01.302739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.302694 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" containerName="payload-processing" Nov 01 05:37:01.302814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.302805 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f2c171-434e-4ebc-a6bd-43eb54eef91c" containerName="payload-processing" Nov 01 05:37:01.306305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.306288 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.364262 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.364213 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb"] Nov 01 05:37:01.383047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.383018 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv76f\" (UniqueName: \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.383204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.383138 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.383316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.383298 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.410350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.410318 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:37:01.410350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.410336 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:37:01.410350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.410318 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:37:01.410862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.410843 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:37:01.484373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.484337 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv76f\" (UniqueName: \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.484565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.484473 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.484684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.484663 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.485134 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.485105 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.487139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.487115 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.547365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.547332 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv76f\" (UniqueName: \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\") pod \"maas-api-revoke-sub-12b9ff7c-zwgcb\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.620566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.620533 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:01.999452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:01.999424 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb"] Nov 01 05:37:02.000196 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:37:02.000169 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b4be78_0bf2_4641_a6a1_d1bc41b3a95f.slice/crio-cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01 WatchSource:0}: Error finding container cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01: Status 404 returned error can't find the container with id cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01 Nov 01 05:37:02.739483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:02.739437 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" event={"ID":"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f","Type":"ContainerStarted","Data":"e11619ea82b5f2d4f6201a7e0b828ad1e03be59cb1b17797605ad261c8c7cfd6"} Nov 01 05:37:02.740020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:02.739544 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" event={"ID":"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f","Type":"ContainerStarted","Data":"cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01"} Nov 01 05:37:02.859725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:02.859670 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" podStartSLOduration=1.8596553949999999 podStartE2EDuration="1.859655395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:37:02.857208437 +0000 UTC m=+2037.997797356" watchObservedRunningTime="2026-11-01 05:37:02.859655395 +0000 UTC m=+2038.000244314" Nov 01 05:37:03.748566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:03.748532 4425 generic.go:356] "Generic (PLEG): container finished" podID="c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" containerID="e11619ea82b5f2d4f6201a7e0b828ad1e03be59cb1b17797605ad261c8c7cfd6" exitCode=0 Nov 01 05:37:03.748993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:03.748626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" event={"ID":"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f","Type":"ContainerDied","Data":"e11619ea82b5f2d4f6201a7e0b828ad1e03be59cb1b17797605ad261c8c7cfd6"} Nov 01 05:37:04.905637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.905608 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:04.938328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.938291 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\") pod \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " Nov 01 05:37:04.938508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.938464 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\") pod \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " Nov 01 05:37:04.938572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.938516 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\" (UniqueName: \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\") pod \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\" (UID: \"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f\") " Nov 01 05:37:04.938829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.938806 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca" pod "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" (UID: "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:37:04.938967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.938953 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:04.940599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.940568 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca" pod "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" (UID: "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:37:04.940711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:04.940664 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f" pod "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" (UID: "c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f"). InnerVolumeSpecName "kube-api-access-xv76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:05.042903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:05.042805 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:05.042903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:05.042846 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv76f\" (UniqueName: \"kubernetes.io/projected/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f-kube-api-access-xv76f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:05.763308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:05.763275 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" Nov 01 05:37:05.763308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:05.763295 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb" event={"ID":"c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f","Type":"ContainerDied","Data":"cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01"} Nov 01 05:37:05.763538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:05.763384 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd06977b85c8e2bf8c786c9e0bae5bd81862fa5a81d1cc573d96d30e17530b01" Nov 01 05:37:06.570345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:06.570315 4425 scope.go:122] "RemoveContainer" containerID="4e7149686957d5eaaff681ac3f0c877599a67a9606625659622f59662c286516" Nov 01 05:37:06.580718 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:06.580694 4425 scope.go:122] "RemoveContainer" containerID="93fa515608fe45f66e48bf815f2edb64c0287690ec6780ed67ec0dc0cd1e35d0" Nov 01 05:37:06.590029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:06.590010 4425 scope.go:122] "RemoveContainer" containerID="71934044f8bb335199b1d4a4965e8654ad2ed6a22e86ad085c4f92a40a74577d" Nov 01 05:37:16.684542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.684506 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn"] Nov 01 05:37:16.685381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.685360 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" containerName="revoke-keys" Nov 01 05:37:16.685381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.685382 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" containerName="revoke-keys" Nov 01 05:37:16.685594 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.685580 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" containerName="revoke-keys" Nov 01 05:37:16.690907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.690886 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.746230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.746195 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn"] Nov 01 05:37:16.792564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.792534 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:37:16.792564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.792549 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:37:16.792794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.792599 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:37:16.795069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.795047 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:37:16.798782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.798761 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.798843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.798812 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827sg\" (UniqueName: \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.799007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.798992 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.900032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.899994 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.900223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.900118 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.900223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.900174 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827sg\" (UniqueName: \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.900769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.900739 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.902897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.902875 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:16.964459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:16.964386 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827sg\" (UniqueName: \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\") pod \"maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:17.006942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:17.006910 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:17.399347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:17.399316 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn"] Nov 01 05:37:17.401189 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:37:17.401158 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b42094_3800_4e06_9852_c99b882e0043.slice/crio-db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7 WatchSource:0}: Error finding container db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7: Status 404 returned error can't find the container with id db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7 Nov 01 05:37:17.856685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:17.856648 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" event={"ID":"a7b42094-3800-4e06-9852-c99b882e0043","Type":"ContainerStarted","Data":"8907831176e3c2595617e98005f0a6ed42fb1fff4db88af5c3884e04e587b28c"} Nov 01 05:37:17.857197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:17.856706 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" event={"ID":"a7b42094-3800-4e06-9852-c99b882e0043","Type":"ContainerStarted","Data":"db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7"} Nov 01 05:37:17.977522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:17.977405 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" podStartSLOduration=1.9773894539999999 podStartE2EDuration="1.977389454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:37:17.974565652 +0000 UTC m=+2053.115154569" watchObservedRunningTime="2026-11-01 05:37:17.977389454 +0000 UTC m=+2053.117978372" Nov 01 05:37:18.865350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:18.865309 4425 generic.go:356] "Generic (PLEG): container finished" podID="a7b42094-3800-4e06-9852-c99b882e0043" containerID="8907831176e3c2595617e98005f0a6ed42fb1fff4db88af5c3884e04e587b28c" exitCode=0 Nov 01 05:37:18.865755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:18.865376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" event={"ID":"a7b42094-3800-4e06-9852-c99b882e0043","Type":"ContainerDied","Data":"8907831176e3c2595617e98005f0a6ed42fb1fff4db88af5c3884e04e587b28c"} Nov 01 05:37:20.029458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.029433 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:20.142424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.142344 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\") pod \"a7b42094-3800-4e06-9852-c99b882e0043\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " Nov 01 05:37:20.142570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.142436 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\" (UniqueName: \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\") pod \"a7b42094-3800-4e06-9852-c99b882e0043\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " Nov 01 05:37:20.142570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.142545 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\") pod \"a7b42094-3800-4e06-9852-c99b882e0043\" (UID: \"a7b42094-3800-4e06-9852-c99b882e0043\") " Nov 01 05:37:20.142768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.142742 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca" pod "a7b42094-3800-4e06-9852-c99b882e0043" (UID: "a7b42094-3800-4e06-9852-c99b882e0043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:37:20.143135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.143035 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:20.144681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.144643 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca" pod "a7b42094-3800-4e06-9852-c99b882e0043" (UID: "a7b42094-3800-4e06-9852-c99b882e0043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:37:20.144777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.144753 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg" pod "a7b42094-3800-4e06-9852-c99b882e0043" (UID: "a7b42094-3800-4e06-9852-c99b882e0043"). InnerVolumeSpecName "kube-api-access-827sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:20.244036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.244004 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b42094-3800-4e06-9852-c99b882e0043-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:20.244036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.244034 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827sg\" (UniqueName: \"kubernetes.io/projected/a7b42094-3800-4e06-9852-c99b882e0043-kube-api-access-827sg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:20.882761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.882732 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" Nov 01 05:37:20.882761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.882755 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn" event={"ID":"a7b42094-3800-4e06-9852-c99b882e0043","Type":"ContainerDied","Data":"db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7"} Nov 01 05:37:20.882976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:20.882802 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db3e9302acfc3a749352bf9facd6534958a4731535f34a95c28739bb6f4be1a7" Nov 01 05:37:24.359881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:24.359844 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:37:24.360571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:24.360065 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" containerID="cri-o://072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec" gracePeriod=30 Nov 01 05:37:26.976474 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:37:26.976434 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:37:28.200820 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:28.200786 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:37:28.310941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:28.310908 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:37:29.777178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.777131 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:37:29.880636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.880593 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881008 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881277 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881456 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:37:29.881589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881462 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881561 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881666 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881720 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881760 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.881908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.881891 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\") pod \"6270326e-b279-4e83-9876-ce4d9ab8af8f\" (UID: \"6270326e-b279-4e83-9876-ce4d9ab8af8f\") " Nov 01 05:37:29.882476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.882453 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.882706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.882683 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:37:29.884319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.884287 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:37:29.884569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.884546 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:37:29.884756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.884734 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:37:29.915919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.905696 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:37:29.915919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.905829 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "kube-api-access-8dm2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:29.915919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.905890 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:37:29.930748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.930713 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token" pod "6270326e-b279-4e83-9876-ce4d9ab8af8f" (UID: "6270326e-b279-4e83-9876-ce4d9ab8af8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984075 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984109 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984125 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dm2g\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-kube-api-access-8dm2g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984143 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984157 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6270326e-b279-4e83-9876-ce4d9ab8af8f-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984175 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984189 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6270326e-b279-4e83-9876-ce4d9ab8af8f-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:29.987332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:29.984202 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6270326e-b279-4e83-9876-ce4d9ab8af8f-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:30.069030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.067728 4425 generic.go:356] "Generic (PLEG): container finished" podID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerID="072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec" exitCode=0 Nov 01 05:37:30.069030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.067820 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" event={"ID":"6270326e-b279-4e83-9876-ce4d9ab8af8f","Type":"ContainerDied","Data":"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec"} Nov 01 05:37:30.069030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.067873 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" event={"ID":"6270326e-b279-4e83-9876-ce4d9ab8af8f","Type":"ContainerDied","Data":"7cbedf4e4e1a8c5f594d934ac6003c8cd51f487f3dbf610628804cc87c0d5c45"} Nov 01 05:37:30.069030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.067903 4425 scope.go:122] "RemoveContainer" containerID="072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec" Nov 01 05:37:30.069030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.068082 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7" Nov 01 05:37:30.136406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.106579 4425 scope.go:122] "RemoveContainer" containerID="072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec" Nov 01 05:37:30.136406 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:37:30.106987 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec\": container with ID starting with 072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec not found: ID does not exist" containerID="072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec" Nov 01 05:37:30.136406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.107025 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec"} err="failed to get container status \"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec\": rpc error: code = NotFound desc = could not find container \"072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec\": container with ID starting with 072d917285adf61bc6dff15b61e98a5375f9cf417af88ab43140a5c63ae37aec not found: ID does not exist" Nov 01 05:37:30.260498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.258463 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:37:30.262783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:30.262748 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-scnj7"] Nov 01 05:37:31.015775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:31.015735 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" path="/var/lib/kubelet/pods/6270326e-b279-4e83-9876-ce4d9ab8af8f/volumes" Nov 01 05:37:31.997278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:31.996525 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kdh6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:37:31.997278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:31.996664 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:37:32.038292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:32.037564 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zzhvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:37:32.038292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:32.037635 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:37:33.201342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.201296 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" podUID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" containerName="payload-processing" containerID="cri-o://fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a" gracePeriod=25 Nov 01 05:37:33.311577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.311533 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" podUID="d11592a8-22a8-44a6-8e05-b04409831f68" containerName="payload-pre-processing" containerID="cri-o://1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b" gracePeriod=25 Nov 01 05:37:33.502792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.502762 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:37:33.596993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.596964 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:37:33.676995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.676964 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\") pod \"d11592a8-22a8-44a6-8e05-b04409831f68\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " Nov 01 05:37:33.677195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677025 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\" (UniqueName: \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\") pod \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " Nov 01 05:37:33.677195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677154 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\" (UniqueName: \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\") pod \"d11592a8-22a8-44a6-8e05-b04409831f68\" (UID: \"d11592a8-22a8-44a6-8e05-b04409831f68\") " Nov 01 05:37:33.677339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677219 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\") pod \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\" (UID: \"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d\") " Nov 01 05:37:33.677557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677402 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume" pod "d11592a8-22a8-44a6-8e05-b04409831f68" (UID: "d11592a8-22a8-44a6-8e05-b04409831f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:37:33.677685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677590 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume" pod "a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" (UID: "a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:37:33.677782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677767 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:33.677822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.677786 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11592a8-22a8-44a6-8e05-b04409831f68-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:33.679525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.679504 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv" pod "a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" (UID: "a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d"). InnerVolumeSpecName "kube-api-access-lwbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:33.679587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.679551 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r" pod "d11592a8-22a8-44a6-8e05-b04409831f68" (UID: "d11592a8-22a8-44a6-8e05-b04409831f68"). InnerVolumeSpecName "kube-api-access-7mj8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:37:33.778578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.778494 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwbjv\" (UniqueName: \"kubernetes.io/projected/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d-kube-api-access-lwbjv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:33.778578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:33.778526 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj8r\" (UniqueName: \"kubernetes.io/projected/d11592a8-22a8-44a6-8e05-b04409831f68-kube-api-access-7mj8r\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:37:34.148920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.148886 4425 generic.go:356] "Generic (PLEG): container finished" podID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" containerID="fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a" exitCode=0 Nov 01 05:37:34.149124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.148936 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" Nov 01 05:37:34.149124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.148972 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" event={"ID":"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d","Type":"ContainerDied","Data":"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a"} Nov 01 05:37:34.149124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.149028 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg" event={"ID":"a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d","Type":"ContainerDied","Data":"f8b2f2fad9a9020caacdff55c3a53ced20f53d7c5b68c4b3267ffe10c2ae1912"} Nov 01 05:37:34.149124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.149052 4425 scope.go:122] "RemoveContainer" containerID="fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a" Nov 01 05:37:34.150837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.150809 4425 generic.go:356] "Generic (PLEG): container finished" podID="d11592a8-22a8-44a6-8e05-b04409831f68" containerID="1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b" exitCode=0 Nov 01 05:37:34.150972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.150847 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" event={"ID":"d11592a8-22a8-44a6-8e05-b04409831f68","Type":"ContainerDied","Data":"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b"} Nov 01 05:37:34.150972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.150858 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" Nov 01 05:37:34.150972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.150894 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj" event={"ID":"d11592a8-22a8-44a6-8e05-b04409831f68","Type":"ContainerDied","Data":"446e8ad99ffb7f30978e0d6878ff28163d9c4a5fb3cda95e218fe027e186940d"} Nov 01 05:37:34.160726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.160642 4425 scope.go:122] "RemoveContainer" containerID="fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a" Nov 01 05:37:34.161066 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:37:34.161037 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a\": container with ID starting with fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a not found: ID does not exist" containerID="fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a" Nov 01 05:37:34.161156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.161077 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a"} err="failed to get container status \"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a\": rpc error: code = NotFound desc = could not find container \"fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a\": container with ID starting with fbe3937e808b9bc9261a4e3553b984c74bebb8b881f9df4221a895dc8431347a not found: ID does not exist" Nov 01 05:37:34.161156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.161109 4425 scope.go:122] "RemoveContainer" containerID="1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b" Nov 01 05:37:34.170319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.170296 4425 scope.go:122] "RemoveContainer" containerID="1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b" Nov 01 05:37:34.170629 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:37:34.170606 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b\": container with ID starting with 1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b not found: ID does not exist" containerID="1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b" Nov 01 05:37:34.170702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.170640 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b"} err="failed to get container status \"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b\": rpc error: code = NotFound desc = could not find container \"1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b\": container with ID starting with 1f4ba975df1ae0cc7932f97a435df278c41d0487bcad21544bc29f56fdefc58b not found: ID does not exist" Nov 01 05:37:34.317276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.317217 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:37:34.368190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.368153 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6c6fdb9-dd8547d47-6z7wg"] Nov 01 05:37:34.479131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.479049 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:37:34.484844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.484811 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6c6fdb9-796cd8c5f9-vrqjj"] Nov 01 05:37:34.982809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.982766 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" path="/var/lib/kubelet/pods/a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d/volumes" Nov 01 05:37:34.983361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:37:34.983345 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11592a8-22a8-44a6-8e05-b04409831f68" path="/var/lib/kubelet/pods/d11592a8-22a8-44a6-8e05-b04409831f68/volumes" Nov 01 05:37:56.948176 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:37:56.948138 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:38:02.492676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:02.492639 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn"] Nov 01 05:38:02.544968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:02.544924 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6c6fdb9-fbrbn"] Nov 01 05:38:02.548094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:02.548064 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:38:02.548354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:02.548328 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" containerID="cri-o://d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a" gracePeriod=30 Nov 01 05:38:02.974443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:02.974412 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b42094-3800-4e06-9852-c99b882e0043" path="/var/lib/kubelet/pods/a7b42094-3800-4e06-9852-c99b882e0043/volumes" Nov 01 05:38:06.640049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:06.640011 4425 scope.go:122] "RemoveContainer" containerID="ad52522251c7b0544a609c7569fa59f3e01483c8ea381d3ab37e1c230e3f986e" Nov 01 05:38:06.649812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:06.649787 4425 scope.go:122] "RemoveContainer" containerID="c428a92f646d582161235ac697cd0258f4b282017380caef38d6868ad8633fa1" Nov 01 05:38:06.660029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:06.660003 4425 scope.go:122] "RemoveContainer" containerID="c9c1b1ff7f0f650548827f4069c2cf13fb03fecdee17ce600f2c8000ed29f2c9" Nov 01 05:38:07.827583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.827517 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:38:07.909026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.908989 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909067 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909167 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909219 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909242 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:38:07.909337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909325 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909378 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909399 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:38:07.909499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909416 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909564 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909594 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\") pod \"0156f43e-648a-4f87-9caa-08f54b6bfb86\" (UID: \"0156f43e-648a-4f87-9caa-08f54b6bfb86\") " Nov 01 05:38:07.909776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909606 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:38:07.909776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909726 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:38:07.909882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.909836 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:38:07.910416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.910390 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:07.910549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.910420 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:07.910549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.910436 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:07.910549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.910451 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0156f43e-648a-4f87-9caa-08f54b6bfb86-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:07.910549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.910466 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:07.911637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.911610 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:38:07.911756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.911670 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:38:07.911756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.911689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "kube-api-access-6d9xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:38:07.911853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:07.911842 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy" pod "0156f43e-648a-4f87-9caa-08f54b6bfb86" (UID: "0156f43e-648a-4f87-9caa-08f54b6bfb86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:38:08.012187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.012156 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:08.012187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.012181 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:08.012187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.012192 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9xg\" (UniqueName: \"kubernetes.io/projected/0156f43e-648a-4f87-9caa-08f54b6bfb86-kube-api-access-6d9xg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:08.012448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.012203 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0156f43e-648a-4f87-9caa-08f54b6bfb86-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:38:08.389740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.389705 4425 generic.go:356] "Generic (PLEG): container finished" podID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerID="d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a" exitCode=0 Nov 01 05:38:08.389922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.389755 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" Nov 01 05:38:08.389922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.389761 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerDied","Data":"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a"} Nov 01 05:38:08.389922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.389809 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp" event={"ID":"0156f43e-648a-4f87-9caa-08f54b6bfb86","Type":"ContainerDied","Data":"712d15415ac55c9aff82f8f48d0631d866e4cca199ebc9c509b02a5238fbeb50"} Nov 01 05:38:08.389922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.389831 4425 scope.go:122] "RemoveContainer" containerID="d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a" Nov 01 05:38:08.401024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.401004 4425 scope.go:122] "RemoveContainer" containerID="b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385" Nov 01 05:38:08.410493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.410476 4425 scope.go:122] "RemoveContainer" containerID="d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a" Nov 01 05:38:08.410775 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:38:08.410757 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a\": container with ID starting with d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a not found: ID does not exist" containerID="d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a" Nov 01 05:38:08.410823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.410784 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a"} err="failed to get container status \"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a\": rpc error: code = NotFound desc = could not find container \"d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a\": container with ID starting with d4348cdf1f41355049edbb5571fa33104ab21655b0812af8053805701afcb39a not found: ID does not exist" Nov 01 05:38:08.410823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.410805 4425 scope.go:122] "RemoveContainer" containerID="b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385" Nov 01 05:38:08.411032 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:38:08.411017 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385\": container with ID starting with b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385 not found: ID does not exist" containerID="b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385" Nov 01 05:38:08.411070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.411036 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385"} err="failed to get container status \"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385\": rpc error: code = NotFound desc = could not find container \"b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385\": container with ID starting with b085ede822e6dd3b567457c5384a079212ed79c4e08a6f50ee0fdec6a5811385 not found: ID does not exist" Nov 01 05:38:08.564863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.564832 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:38:08.615442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.615410 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c6c6fdb9-gw-openshift-default-78784b5865-wt8rp"] Nov 01 05:38:08.974309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:08.974280 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" path="/var/lib/kubelet/pods/0156f43e-648a-4f87-9caa-08f54b6bfb86/volumes" Nov 01 05:38:26.948393 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:38:26.948310 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:38:51.455318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455282 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:38:51.455866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455851 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.455914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455867 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.455914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455889 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.455914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455894 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.455914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455904 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" Nov 01 05:38:51.455914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455910 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455919 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" containerName="payload-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455924 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" containerName="payload-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455938 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a7b42094-3800-4e06-9852-c99b882e0043" containerName="revoke-keys" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455943 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b42094-3800-4e06-9852-c99b882e0043" containerName="revoke-keys" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455959 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d11592a8-22a8-44a6-8e05-b04409831f68" containerName="payload-pre-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.455964 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11592a8-22a8-44a6-8e05-b04409831f68" containerName="payload-pre-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456048 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11592a8-22a8-44a6-8e05-b04409831f68" containerName="payload-pre-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456060 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456067 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0156f43e-648a-4f87-9caa-08f54b6bfb86" containerName="istio-proxy" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456074 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6270326e-b279-4e83-9876-ce4d9ab8af8f" containerName="istio-proxy" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456080 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92e707b-1f6a-4f43-ae91-bfd1a4e3dc5d" containerName="payload-processing" Nov 01 05:38:51.456088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.456086 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b42094-3800-4e06-9852-c99b882e0043" containerName="revoke-keys" Nov 01 05:38:51.459622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.459605 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.566277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.564071 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2q6\" (UniqueName: \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.566277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.564317 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.567580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.567555 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a08eb536-dockercfg-bpthx\"" Nov 01 05:38:51.567832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.567791 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a08eb536\"" Nov 01 05:38:51.570181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.570157 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:38:51.666209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.666173 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2q6\" (UniqueName: \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.666517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.666408 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.667413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.667362 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.728959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.728884 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2q6\" (UniqueName: \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\") pod \"payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:51.773595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:51.773565 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:52.168665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:52.152969 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:38:52.168665 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:38:52.166628 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d76782_0b88_4948_b199_5a12d13da32b.slice/crio-3606fb0336e66b148defe3fbac45f7c9dc1431ce93fba6f52a9d1675b004cdfa WatchSource:0}: Error finding container 3606fb0336e66b148defe3fbac45f7c9dc1431ce93fba6f52a9d1675b004cdfa: Status 404 returned error can't find the container with id 3606fb0336e66b148defe3fbac45f7c9dc1431ce93fba6f52a9d1675b004cdfa Nov 01 05:38:52.680987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:52.680949 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" event={"ID":"f6d76782-0b88-4948-b199-5a12d13da32b","Type":"ContainerStarted","Data":"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600"} Nov 01 05:38:52.681441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:52.681018 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:38:52.681441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:52.681040 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" event={"ID":"f6d76782-0b88-4948-b199-5a12d13da32b","Type":"ContainerStarted","Data":"3606fb0336e66b148defe3fbac45f7c9dc1431ce93fba6f52a9d1675b004cdfa"} Nov 01 05:38:52.801504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:38:52.801453 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" podStartSLOduration=1.8014409580000001 podStartE2EDuration="1.801440958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:38:52.798384497 +0000 UTC m=+2147.938973443" watchObservedRunningTime="2026-11-01 05:38:52.801440958 +0000 UTC m=+2147.942029899" Nov 01 05:38:56.950771 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:38:56.950612 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:39:03.690898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:03.690856 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:39:26.948680 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:39:26.948638 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:39:32.033305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.033242 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:32.037580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.037559 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.094805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.094771 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:32.136647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.136614 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4"] Nov 01 05:39:32.140935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.140912 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.168544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.168508 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2g4\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.168708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.168554 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.168708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.168657 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.168805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.168708 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.194117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.194090 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:39:32.194321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.194304 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:39:32.194412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.194392 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:39:32.194454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.194435 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:39:32.205684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.205663 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4"] Nov 01 05:39:32.270235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270198 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.270235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270240 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.270523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270397 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.270523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270477 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.270523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270514 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ng5r\" (UniqueName: \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.270692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270575 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.270692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.270684 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2g4\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.271571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.271536 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.271900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.271838 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.283610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.283543 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.333559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.333527 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2g4\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.350982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.350953 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:32.371879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.371849 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.371996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.371969 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.372121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.372106 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ng5r\" (UniqueName: \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.372677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.372647 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.374400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.374382 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.436972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.436935 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ng5r\" (UniqueName: \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\") pod \"maas-api-revoke-sub-1a50973c-pcbc4\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.454541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.454513 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:32.734923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.734891 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:32.737017 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:39:32.736989 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fe5aeb_8313_480d_985d_d51d22807386.slice/crio-0b2db1b454d12c98734e9faf058c818b15088d2cf02a8a941a09f420162fbce2 WatchSource:0}: Error finding container 0b2db1b454d12c98734e9faf058c818b15088d2cf02a8a941a09f420162fbce2: Status 404 returned error can't find the container with id 0b2db1b454d12c98734e9faf058c818b15088d2cf02a8a941a09f420162fbce2 Nov 01 05:39:32.847992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.847918 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4"] Nov 01 05:39:32.850613 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:39:32.850585 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5368d7e_9aaf_4e5e_8429_299d89c55c86.slice/crio-05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4 WatchSource:0}: Error finding container 05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4: Status 404 returned error can't find the container with id 05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4 Nov 01 05:39:32.954664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.954626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" event={"ID":"b5368d7e-9aaf-4e5e-8429-299d89c55c86","Type":"ContainerStarted","Data":"05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4"} Nov 01 05:39:32.956516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.956488 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" event={"ID":"56fe5aeb-8313-480d-985d-d51d22807386","Type":"ContainerStarted","Data":"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984"} Nov 01 05:39:32.956651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.956536 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" event={"ID":"56fe5aeb-8313-480d-985d-d51d22807386","Type":"ContainerStarted","Data":"0b2db1b454d12c98734e9faf058c818b15088d2cf02a8a941a09f420162fbce2"} Nov 01 05:39:32.956651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:32.956578 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:33.076599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:33.076543 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" podStartSLOduration=2.076526369 podStartE2EDuration="2.076526369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:39:33.074939355 +0000 UTC m=+2188.215528302" watchObservedRunningTime="2026-11-01 05:39:33.076526369 +0000 UTC m=+2188.217115304" Nov 01 05:39:33.965605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:33.965564 4425 generic.go:356] "Generic (PLEG): container finished" podID="b5368d7e-9aaf-4e5e-8429-299d89c55c86" containerID="5b64c569609186243d689193f539dc034457a7b919026d53920064e75b8ab0cf" exitCode=0 Nov 01 05:39:33.965803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:33.965641 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" event={"ID":"b5368d7e-9aaf-4e5e-8429-299d89c55c86","Type":"ContainerDied","Data":"5b64c569609186243d689193f539dc034457a7b919026d53920064e75b8ab0cf"} Nov 01 05:39:35.136341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.136314 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:35.308612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.308538 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\") pod \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " Nov 01 05:39:35.308612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.308601 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\" (UniqueName: \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\") pod \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " Nov 01 05:39:35.308820 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.308627 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\") pod \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\" (UID: \"b5368d7e-9aaf-4e5e-8429-299d89c55c86\") " Nov 01 05:39:35.309012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.308985 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca" pod "b5368d7e-9aaf-4e5e-8429-299d89c55c86" (UID: "b5368d7e-9aaf-4e5e-8429-299d89c55c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:39:35.309162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.309130 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:35.311470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.311447 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca" pod "b5368d7e-9aaf-4e5e-8429-299d89c55c86" (UID: "b5368d7e-9aaf-4e5e-8429-299d89c55c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:39:35.311470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.311461 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r" pod "b5368d7e-9aaf-4e5e-8429-299d89c55c86" (UID: "b5368d7e-9aaf-4e5e-8429-299d89c55c86"). InnerVolumeSpecName "kube-api-access-8ng5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:39:35.410588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.410555 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ng5r\" (UniqueName: \"kubernetes.io/projected/b5368d7e-9aaf-4e5e-8429-299d89c55c86-kube-api-access-8ng5r\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:35.410588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.410584 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5368d7e-9aaf-4e5e-8429-299d89c55c86-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:35.988795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.988753 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" Nov 01 05:39:35.989023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.988797 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4" event={"ID":"b5368d7e-9aaf-4e5e-8429-299d89c55c86","Type":"ContainerDied","Data":"05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4"} Nov 01 05:39:35.989023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:35.988901 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05066a857bb198b547674c92cf9b74d0ea7bedbf5714887f4061bd352ee910d4" Nov 01 05:39:42.846639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.846604 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5"] Nov 01 05:39:42.847468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.847446 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b5368d7e-9aaf-4e5e-8429-299d89c55c86" containerName="revoke-keys" Nov 01 05:39:42.847468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.847468 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5368d7e-9aaf-4e5e-8429-299d89c55c86" containerName="revoke-keys" Nov 01 05:39:42.847656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.847609 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5368d7e-9aaf-4e5e-8429-299d89c55c86" containerName="revoke-keys" Nov 01 05:39:42.851385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.851364 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:42.900606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.900571 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sdn\" (UniqueName: \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:42.900766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.900715 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:42.900910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.900886 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:42.907509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.907482 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5"] Nov 01 05:39:42.961027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.960999 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:39:42.961610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.961591 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:39:42.961878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.961856 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:39:42.962083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:42.961919 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:39:43.009549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.009508 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sdn\" (UniqueName: \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.009748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.009680 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.009839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.009821 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.011012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.010968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.014607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.014579 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.072017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.071985 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sdn\" (UniqueName: \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\") pod \"maas-api-revoke-sub-6dcfd929-l6xw5\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.166210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.166125 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:43.544458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.544425 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5"] Nov 01 05:39:43.545319 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:39:43.545286 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688126f6_7937_42d4_9c4d_3f408b0ee0ea.slice/crio-1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1 WatchSource:0}: Error finding container 1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1: Status 404 returned error can't find the container with id 1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1 Nov 01 05:39:43.969028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:43.968997 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:39:44.071743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:44.070723 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" event={"ID":"688126f6-7937-42d4-9c4d-3f408b0ee0ea","Type":"ContainerStarted","Data":"de93cd8fc643b73ee756f80d2f4543c5fdb3e71bcbf060a98ac1274af9471889"} Nov 01 05:39:44.071743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:44.070788 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" event={"ID":"688126f6-7937-42d4-9c4d-3f408b0ee0ea","Type":"ContainerStarted","Data":"1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1"} Nov 01 05:39:44.189416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:44.189352 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" podStartSLOduration=2.18933789 podStartE2EDuration="2.18933789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:39:44.189088018 +0000 UTC m=+2199.329676949" watchObservedRunningTime="2026-11-01 05:39:44.18933789 +0000 UTC m=+2199.329926809" Nov 01 05:39:44.242970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:44.242896 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:44.244492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:44.244374 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:39:45.082228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:45.082197 4425 generic.go:356] "Generic (PLEG): container finished" podID="688126f6-7937-42d4-9c4d-3f408b0ee0ea" containerID="de93cd8fc643b73ee756f80d2f4543c5fdb3e71bcbf060a98ac1274af9471889" exitCode=0 Nov 01 05:39:45.082669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:45.082294 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" event={"ID":"688126f6-7937-42d4-9c4d-3f408b0ee0ea","Type":"ContainerDied","Data":"de93cd8fc643b73ee756f80d2f4543c5fdb3e71bcbf060a98ac1274af9471889"} Nov 01 05:39:46.243243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.243215 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:46.361374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.361218 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\" (UniqueName: \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\") pod \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " Nov 01 05:39:46.361619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.361514 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\") pod \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " Nov 01 05:39:46.361619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.361561 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\") pod \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\" (UID: \"688126f6-7937-42d4-9c4d-3f408b0ee0ea\") " Nov 01 05:39:46.362379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.362351 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca" pod "688126f6-7937-42d4-9c4d-3f408b0ee0ea" (UID: "688126f6-7937-42d4-9c4d-3f408b0ee0ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:39:46.364615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.364586 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn" pod "688126f6-7937-42d4-9c4d-3f408b0ee0ea" (UID: "688126f6-7937-42d4-9c4d-3f408b0ee0ea"). InnerVolumeSpecName "kube-api-access-r4sdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:39:46.366714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.366688 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca" pod "688126f6-7937-42d4-9c4d-3f408b0ee0ea" (UID: "688126f6-7937-42d4-9c4d-3f408b0ee0ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:39:46.463667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.463634 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:46.463667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.463663 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688126f6-7937-42d4-9c4d-3f408b0ee0ea-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:46.463667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:46.463675 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4sdn\" (UniqueName: \"kubernetes.io/projected/688126f6-7937-42d4-9c4d-3f408b0ee0ea-kube-api-access-r4sdn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:47.096481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:47.096446 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" event={"ID":"688126f6-7937-42d4-9c4d-3f408b0ee0ea","Type":"ContainerDied","Data":"1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1"} Nov 01 05:39:47.096481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:47.096480 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5" Nov 01 05:39:47.096697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:47.096503 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1230b49c287b2212e6e69348ee5181139b5904a9c027f25c969bdde705a7c6c1" Nov 01 05:39:49.243915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.243875 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" containerID="cri-o://16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984" gracePeriod=25 Nov 01 05:39:49.520313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.520282 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:49.606846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.606813 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\") pod \"56fe5aeb-8313-480d-985d-d51d22807386\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " Nov 01 05:39:49.607052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.606864 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\") pod \"56fe5aeb-8313-480d-985d-d51d22807386\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " Nov 01 05:39:49.607052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.606952 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\") pod \"56fe5aeb-8313-480d-985d-d51d22807386\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " Nov 01 05:39:49.607052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.607010 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\") pod \"56fe5aeb-8313-480d-985d-d51d22807386\" (UID: \"56fe5aeb-8313-480d-985d-d51d22807386\") " Nov 01 05:39:49.607351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.607321 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume" pod "56fe5aeb-8313-480d-985d-d51d22807386" (UID: "56fe5aeb-8313-480d-985d-d51d22807386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:39:49.607438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.607380 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume" pod "56fe5aeb-8313-480d-985d-d51d22807386" (UID: "56fe5aeb-8313-480d-985d-d51d22807386"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:39:49.607627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.607609 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:49.607678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.607633 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fe5aeb-8313-480d-985d-d51d22807386-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:49.609435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.609412 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials" pod "56fe5aeb-8313-480d-985d-d51d22807386" (UID: "56fe5aeb-8313-480d-985d-d51d22807386"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:39:49.609532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.609444 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4" pod "56fe5aeb-8313-480d-985d-d51d22807386" (UID: "56fe5aeb-8313-480d-985d-d51d22807386"). InnerVolumeSpecName "kube-api-access-qt2g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:39:49.708655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.708623 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:49.708655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:49.708658 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2g4\" (UniqueName: \"kubernetes.io/projected/56fe5aeb-8313-480d-985d-d51d22807386-kube-api-access-qt2g4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:39:50.117453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.117407 4425 generic.go:356] "Generic (PLEG): container finished" podID="56fe5aeb-8313-480d-985d-d51d22807386" containerID="16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984" exitCode=0 Nov 01 05:39:50.117635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.117462 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" Nov 01 05:39:50.117635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.117495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" event={"ID":"56fe5aeb-8313-480d-985d-d51d22807386","Type":"ContainerDied","Data":"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984"} Nov 01 05:39:50.117635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.117553 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4" event={"ID":"56fe5aeb-8313-480d-985d-d51d22807386","Type":"ContainerDied","Data":"0b2db1b454d12c98734e9faf058c818b15088d2cf02a8a941a09f420162fbce2"} Nov 01 05:39:50.117635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.117574 4425 scope.go:122] "RemoveContainer" containerID="16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984" Nov 01 05:39:50.131401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.131034 4425 scope.go:122] "RemoveContainer" containerID="16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984" Nov 01 05:39:50.131498 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:39:50.131477 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984\": container with ID starting with 16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984 not found: ID does not exist" containerID="16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984" Nov 01 05:39:50.131556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.131507 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984"} err="failed to get container status \"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984\": rpc error: code = NotFound desc = could not find container \"16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984\": container with ID starting with 16022aa96246e008e0f5b467e265bf308ec45fbb57a0de39420084a47dfa7984 not found: ID does not exist" Nov 01 05:39:50.298293 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.298246 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:50.355303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.353072 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5sc7v4"] Nov 01 05:39:50.973870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:50.973824 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fe5aeb-8313-480d-985d-d51d22807386" path="/var/lib/kubelet/pods/56fe5aeb-8313-480d-985d-d51d22807386/volumes" Nov 01 05:39:54.165495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.165461 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:39:54.166166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166150 4425 cpu_manager.go:414] "containerMap: removing container" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" Nov 01 05:39:54.166212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166168 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" Nov 01 05:39:54.166212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166192 4425 cpu_manager.go:414] "containerMap: removing container" podUID="688126f6-7937-42d4-9c4d-3f408b0ee0ea" containerName="revoke-keys" Nov 01 05:39:54.166212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166198 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688126f6-7937-42d4-9c4d-3f408b0ee0ea" containerName="revoke-keys" Nov 01 05:39:54.166354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166321 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fe5aeb-8313-480d-985d-d51d22807386" containerName="payload-processing" Nov 01 05:39:54.166354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.166336 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="688126f6-7937-42d4-9c4d-3f408b0ee0ea" containerName="revoke-keys" Nov 01 05:39:54.171072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.171050 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.227706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.227673 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:39:54.268743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.268706 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.268918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.268774 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9c5r\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.268918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.268873 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.269050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.269037 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.274432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.274406 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-f16ea820c5f3\"" Nov 01 05:39:54.370734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.370702 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.370931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.370745 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.370931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.370800 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9c5r\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.370931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.370841 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.371567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.371545 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.371680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.371659 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.373597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.373579 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.433370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.433302 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9c5r\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.484960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.484922 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:54.876552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:54.876397 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:39:54.880222 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:39:54.880181 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7d9053_976a_4578_b5c5_470fa5966aea.slice/crio-a2dcb6391ba62ca20ff6958d5414e94370c272f0ce84e95902ecaf6418422757 WatchSource:0}: Error finding container a2dcb6391ba62ca20ff6958d5414e94370c272f0ce84e95902ecaf6418422757: Status 404 returned error can't find the container with id a2dcb6391ba62ca20ff6958d5414e94370c272f0ce84e95902ecaf6418422757 Nov 01 05:39:55.152952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:55.152853 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" event={"ID":"4e7d9053-976a-4578-b5c5-470fa5966aea","Type":"ContainerStarted","Data":"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4"} Nov 01 05:39:55.152952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:55.152905 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" event={"ID":"4e7d9053-976a-4578-b5c5-470fa5966aea","Type":"ContainerStarted","Data":"a2dcb6391ba62ca20ff6958d5414e94370c272f0ce84e95902ecaf6418422757"} Nov 01 05:39:55.153198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:55.152960 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:39:55.273368 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:55.273312 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" podStartSLOduration=1.273296486 podStartE2EDuration="1.273296486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:39:55.270894772 +0000 UTC m=+2210.411483691" watchObservedRunningTime="2026-11-01 05:39:55.273296486 +0000 UTC m=+2210.413885404" Nov 01 05:39:56.948575 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:39:56.948536 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:39:59.813819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:39:59.813786 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:40:06.056674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.056642 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n"] Nov 01 05:40:06.061304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.061284 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.115393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.115355 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.115597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.115491 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.115597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.115526 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsq9\" (UniqueName: \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.118660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.118636 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n"] Nov 01 05:40:06.163273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.163221 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:40:06.165487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.165461 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:40:06.165487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.165467 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:40:06.165669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.165518 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:40:06.165669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.165518 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:40:06.217314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.217244 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.217519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.217371 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.217519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.217403 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsq9\" (UniqueName: \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.218139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.218062 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.219876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.219857 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.283690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.283652 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsq9\" (UniqueName: \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\") pod \"maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.375012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.374976 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:06.379315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.379288 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:40:06.762588 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:06.762558 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a290ea0_1c09_4fb2_9edc_b1e73a1f985f.slice/crio-a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59 WatchSource:0}: Error finding container a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59: Status 404 returned error can't find the container with id a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59 Nov 01 05:40:06.763870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:06.763848 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n"] Nov 01 05:40:07.235839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:07.235800 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" event={"ID":"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f","Type":"ContainerStarted","Data":"7ac86a166267c9135dc1fbc9011963f32d806833da4fb355660b9c8eb1dad7c5"} Nov 01 05:40:07.236225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:07.235854 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" event={"ID":"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f","Type":"ContainerStarted","Data":"a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59"} Nov 01 05:40:07.354961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:07.354888 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" podStartSLOduration=1.3548738980000001 podStartE2EDuration="1.354873898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:07.352721764 +0000 UTC m=+2222.493310715" watchObservedRunningTime="2026-11-01 05:40:07.354873898 +0000 UTC m=+2222.495462842" Nov 01 05:40:08.243922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:08.243894 4425 generic.go:356] "Generic (PLEG): container finished" podID="7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" containerID="7ac86a166267c9135dc1fbc9011963f32d806833da4fb355660b9c8eb1dad7c5" exitCode=0 Nov 01 05:40:08.244356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:08.243985 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" event={"ID":"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f","Type":"ContainerDied","Data":"7ac86a166267c9135dc1fbc9011963f32d806833da4fb355660b9c8eb1dad7c5"} Nov 01 05:40:09.411892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.411863 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:09.456824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.456795 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\") pod \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " Nov 01 05:40:09.457000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.456883 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\" (UniqueName: \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\") pod \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " Nov 01 05:40:09.457000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.456955 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\") pod \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\" (UID: \"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f\") " Nov 01 05:40:09.457408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.457378 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca" pod "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" (UID: "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:40:09.457653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.457600 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:09.459357 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.459328 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca" pod "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" (UID: "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:40:09.459468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.459449 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9" pod "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" (UID: "7a290ea0-1c09-4fb2-9edc-b1e73a1f985f"). InnerVolumeSpecName "kube-api-access-xjsq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:09.559021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.558914 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:09.559021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:09.558947 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsq9\" (UniqueName: \"kubernetes.io/projected/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f-kube-api-access-xjsq9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:10.259441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:10.259406 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" Nov 01 05:40:10.259441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:10.259432 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n" event={"ID":"7a290ea0-1c09-4fb2-9edc-b1e73a1f985f","Type":"ContainerDied","Data":"a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59"} Nov 01 05:40:10.259661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:10.259478 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53796681a13fec2e8a293cd2a93c59d78c7b6d1d963c4641d87328f3e547a59" Nov 01 05:40:11.380093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.380050 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" podUID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" containerName="payload-processing" containerID="cri-o://e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101" gracePeriod=25 Nov 01 05:40:11.672996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.672961 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:40:11.783042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783006 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\") pod \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " Nov 01 05:40:11.783234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783149 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\") pod \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " Nov 01 05:40:11.783234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783180 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\") pod \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " Nov 01 05:40:11.783375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783232 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\") pod \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\" (UID: \"44f8b27a-a55e-466b-8895-e4b6aaf503f1\") " Nov 01 05:40:11.783580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783549 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume" pod "44f8b27a-a55e-466b-8895-e4b6aaf503f1" (UID: "44f8b27a-a55e-466b-8895-e4b6aaf503f1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:40:11.783649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783612 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume" pod "44f8b27a-a55e-466b-8895-e4b6aaf503f1" (UID: "44f8b27a-a55e-466b-8895-e4b6aaf503f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:40:11.783883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783864 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:11.783946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.783884 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f8b27a-a55e-466b-8895-e4b6aaf503f1-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:11.785388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.785366 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g" pod "44f8b27a-a55e-466b-8895-e4b6aaf503f1" (UID: "44f8b27a-a55e-466b-8895-e4b6aaf503f1"). InnerVolumeSpecName "kube-api-access-5q66g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:11.785494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.785483 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials" pod "44f8b27a-a55e-466b-8895-e4b6aaf503f1" (UID: "44f8b27a-a55e-466b-8895-e4b6aaf503f1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:11.884797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.884764 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q66g\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-kube-api-access-5q66g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:11.884797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:11.884796 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44f8b27a-a55e-466b-8895-e4b6aaf503f1-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:12.273622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.273589 4425 generic.go:356] "Generic (PLEG): container finished" podID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" containerID="e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101" exitCode=0 Nov 01 05:40:12.273838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.273636 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" Nov 01 05:40:12.273838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.273681 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" event={"ID":"44f8b27a-a55e-466b-8895-e4b6aaf503f1","Type":"ContainerDied","Data":"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101"} Nov 01 05:40:12.273838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.273744 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx" event={"ID":"44f8b27a-a55e-466b-8895-e4b6aaf503f1","Type":"ContainerDied","Data":"e8e99e6275114c4b2fe5a28d9f345664fea2e3933ddd20f3c80e87134aa10c7d"} Nov 01 05:40:12.273838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.273776 4425 scope.go:122] "RemoveContainer" containerID="e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101" Nov 01 05:40:12.284169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.284152 4425 scope.go:122] "RemoveContainer" containerID="e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101" Nov 01 05:40:12.284481 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:40:12.284458 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101\": container with ID starting with e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101 not found: ID does not exist" containerID="e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101" Nov 01 05:40:12.284555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.284491 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101"} err="failed to get container status \"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101\": rpc error: code = NotFound desc = could not find container \"e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101\": container with ID starting with e9d0d8f05720831bfb0f0ab231fb8b9fc46ba5d4d666ea878dbd5e1925c41101 not found: ID does not exist" Nov 01 05:40:12.442850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.442817 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:40:12.452102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.452064 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-7fbcjj8hx"] Nov 01 05:40:12.973850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:12.973818 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" path="/var/lib/kubelet/pods/44f8b27a-a55e-466b-8895-e4b6aaf503f1/volumes" Nov 01 05:40:14.835230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:14.835183 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" podUID="0da5c379-5397-4727-ad13-1f4450693c83" containerName="main" containerID="cri-o://3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9" gracePeriod=45 Nov 01 05:40:14.859724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:14.859694 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" podUID="0da5c379-5397-4727-ad13-1f4450693c83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused" Nov 01 05:40:15.113588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.113564 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:40:15.226544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226511 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\" (UniqueName: \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.226544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226562 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\" (UniqueName: \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.226812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226673 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.226812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226690 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.226812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226727 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.226812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226762 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\") pod \"0da5c379-5397-4727-ad13-1f4450693c83\" (UID: \"0da5c379-5397-4727-ad13-1f4450693c83\") " Nov 01 05:40:15.227028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.226978 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:15.227095 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.227065 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:15.227095 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.227084 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:15.227411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.227363 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.227411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.227384 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.227411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.227401 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.229152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.229127 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "kube-api-access-mbqg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:15.229152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.229130 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:15.229320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.229178 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs" pod "0da5c379-5397-4727-ad13-1f4450693c83" (UID: "0da5c379-5397-4727-ad13-1f4450693c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:40:15.295485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.295453 4425 generic.go:356] "Generic (PLEG): container finished" podID="0da5c379-5397-4727-ad13-1f4450693c83" containerID="3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9" exitCode=0 Nov 01 05:40:15.295622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.295499 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" Nov 01 05:40:15.295622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.295511 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" event={"ID":"0da5c379-5397-4727-ad13-1f4450693c83","Type":"ContainerDied","Data":"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9"} Nov 01 05:40:15.295622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.295544 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m" event={"ID":"0da5c379-5397-4727-ad13-1f4450693c83","Type":"ContainerDied","Data":"b63b74d1340d3051faba56ebbf74016378e765079310a3e72dd429a6ff533bcd"} Nov 01 05:40:15.295622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.295562 4425 scope.go:122] "RemoveContainer" containerID="3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9" Nov 01 05:40:15.305864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.305843 4425 scope.go:122] "RemoveContainer" containerID="3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9" Nov 01 05:40:15.306123 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:40:15.306102 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9\": container with ID starting with 3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9 not found: ID does not exist" containerID="3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9" Nov 01 05:40:15.306173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.306132 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9"} err="failed to get container status \"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9\": rpc error: code = NotFound desc = could not find container \"3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9\": container with ID starting with 3f9cbd85e11cdcd8dbb312fed9e8df3b337b7f30f5ce2faff3715f017f92d6e9 not found: ID does not exist" Nov 01 05:40:15.329014 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.328982 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbqg6\" (UniqueName: \"kubernetes.io/projected/0da5c379-5397-4727-ad13-1f4450693c83-kube-api-access-mbqg6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.329142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.329019 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da5c379-5397-4727-ad13-1f4450693c83-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.329142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.329036 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da5c379-5397-4727-ad13-1f4450693c83-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:15.463298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.463188 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:40:15.469800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:15.469765 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/auth-test-model-afc555-kserve-7546699ddf-gml4m"] Nov 01 05:40:16.974282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:16.974223 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da5c379-5397-4727-ad13-1f4450693c83" path="/var/lib/kubelet/pods/0da5c379-5397-4727-ad13-1f4450693c83/volumes" Nov 01 05:40:24.047968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:24.047934 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr"] Nov 01 05:40:24.098398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:24.098370 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cda842-4dgrr"] Nov 01 05:40:24.974118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:24.974080 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33398d25-592f-44cd-bd27-162e8c6f1ec4" path="/var/lib/kubelet/pods/33398d25-592f-44cd-bd27-162e8c6f1ec4/volumes" Nov 01 05:40:26.539399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:26.539363 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:40:26.947656 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:40:26.947619 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:40:32.386451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.386414 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:40:32.387005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.386991 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0da5c379-5397-4727-ad13-1f4450693c83" containerName="main" Nov 01 05:40:32.387051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387006 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da5c379-5397-4727-ad13-1f4450693c83" containerName="main" Nov 01 05:40:32.387051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387022 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" containerName="revoke-keys" Nov 01 05:40:32.387051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387027 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" containerName="revoke-keys" Nov 01 05:40:32.387051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387047 4425 cpu_manager.go:414] "containerMap: removing container" podUID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" containerName="payload-processing" Nov 01 05:40:32.387181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387053 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" containerName="payload-processing" Nov 01 05:40:32.387181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387148 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f8b27a-a55e-466b-8895-e4b6aaf503f1" containerName="payload-processing" Nov 01 05:40:32.387181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387163 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da5c379-5397-4727-ad13-1f4450693c83" containerName="main" Nov 01 05:40:32.387181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.387170 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" containerName="revoke-keys" Nov 01 05:40:32.390854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.390832 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.445603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.444891 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-faa8ff-openshift-default-dockercfg-bl4m9\"" Nov 01 05:40:32.446942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.446875 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447175 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447388 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447570 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447727 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447848 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.447996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.447981 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.448127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.448111 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwv7\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.448267 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.448231 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.524059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.523660 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549021 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549101 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549121 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549153 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549180 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwv7\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549202 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549327 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549391 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549446 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.549809 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.550281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.550098 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.551039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.550579 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.551039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.550861 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.551297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.551266 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.554501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.554474 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.556721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.556695 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.613568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.613535 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.613729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.613670 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwv7\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\") pod \"e2e-api-a-faa8ff-openshift-default-667f695f57-982wc\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:32.705295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:32.705201 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:33.144795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.144768 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:40:33.145534 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:33.145503 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990e9ab6_e3b5_4ead_a448_4fc047650186.slice/crio-f39f1eb5ff4bf45b27e611513c228be26872ce2c28868a42a0c7065b13177331 WatchSource:0}: Error finding container f39f1eb5ff4bf45b27e611513c228be26872ce2c28868a42a0c7065b13177331: Status 404 returned error can't find the container with id f39f1eb5ff4bf45b27e611513c228be26872ce2c28868a42a0c7065b13177331 Nov 01 05:40:33.147991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.147957 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:33.148065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.148026 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:33.148065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.148056 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:33.419623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.419494 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerStarted","Data":"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2"} Nov 01 05:40:33.420079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.419643 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerStarted","Data":"f39f1eb5ff4bf45b27e611513c228be26872ce2c28868a42a0c7065b13177331"} Nov 01 05:40:33.549246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.549194 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podStartSLOduration=1.5491766249999999 podStartE2EDuration="1.549176625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:33.54606921 +0000 UTC m=+2248.686658156" watchObservedRunningTime="2026-11-01 05:40:33.549176625 +0000 UTC m=+2248.689765544" Nov 01 05:40:33.705438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:33.705340 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:34.712759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:34.712732 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:35.431533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:35.431499 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:35.432542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:35.432515 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:40:40.207314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.204949 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:40:40.231842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.231770 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.254449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.254421 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:40:40.298895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.298866 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-faa8ff\"" Nov 01 05:40:40.299080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.298993 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-faa8ff-dockercfg-w4wtx\"" Nov 01 05:40:40.349103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.349069 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.349269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.349133 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps7m\" (UniqueName: \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.363269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.363217 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:40:40.367465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.367448 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.424265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.424218 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:40:40.450780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.450740 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.450951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.450900 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.450999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.450980 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps7m\" (UniqueName: \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.451052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.451024 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mds\" (UniqueName: \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.451631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.451614 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.514792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.514715 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps7m\" (UniqueName: \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\") pod \"payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.552125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.552092 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mds\" (UniqueName: \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.552318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.552215 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:40.552318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.552238 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.552946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.552927 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.614416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.614126 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mds\" (UniqueName: \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\") pod \"payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.687830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.687784 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:40.933639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:40.933606 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:40:40.934127 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:40.934104 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb04c9c_5aa3_4b56_93d5_fa15e06a488f.slice/crio-3af9970e0e56d387adde38439776f4a6b08be8a87ff2dc6a81cfb823aff6b0cb WatchSource:0}: Error finding container 3af9970e0e56d387adde38439776f4a6b08be8a87ff2dc6a81cfb823aff6b0cb: Status 404 returned error can't find the container with id 3af9970e0e56d387adde38439776f4a6b08be8a87ff2dc6a81cfb823aff6b0cb Nov 01 05:40:41.075488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.075458 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:40:41.076067 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:41.076038 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872cce58_de80_4040_a73a_43f828fac090.slice/crio-58fc6d170e803f83454f019801b03b02d9c0dc9c769112eaa13ee3b8f8d1f194 WatchSource:0}: Error finding container 58fc6d170e803f83454f019801b03b02d9c0dc9c769112eaa13ee3b8f8d1f194: Status 404 returned error can't find the container with id 58fc6d170e803f83454f019801b03b02d9c0dc9c769112eaa13ee3b8f8d1f194 Nov 01 05:40:41.474269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.474225 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" event={"ID":"872cce58-de80-4040-a73a-43f828fac090","Type":"ContainerStarted","Data":"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf"} Nov 01 05:40:41.474758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.474313 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" event={"ID":"872cce58-de80-4040-a73a-43f828fac090","Type":"ContainerStarted","Data":"58fc6d170e803f83454f019801b03b02d9c0dc9c769112eaa13ee3b8f8d1f194"} Nov 01 05:40:41.474758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.474344 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:41.476006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.475981 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" event={"ID":"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f","Type":"ContainerStarted","Data":"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e"} Nov 01 05:40:41.476121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.476037 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" event={"ID":"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f","Type":"ContainerStarted","Data":"3af9970e0e56d387adde38439776f4a6b08be8a87ff2dc6a81cfb823aff6b0cb"} Nov 01 05:40:41.476121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.476060 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:41.560753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.560711 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" podUID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" containerName="main" containerID="cri-o://f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583" gracePeriod=45 Nov 01 05:40:41.592883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.592837 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" podStartSLOduration=1.592819701 podStartE2EDuration="1.592819701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:41.591880879 +0000 UTC m=+2256.732469799" watchObservedRunningTime="2026-11-01 05:40:41.592819701 +0000 UTC m=+2256.733408619" Nov 01 05:40:41.865571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.865546 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:40:41.969417 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969389 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969426 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969507 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969525 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969562 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969586 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\" (UniqueName: \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\") pod \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\" (UID: \"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4\") " Nov 01 05:40:41.969840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969735 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:41.969840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969769 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:41.969982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.969912 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:41.970080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.970064 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:41.970139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.970082 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:41.970139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.970094 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:41.971805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.971785 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:40:41.971916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.971831 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:40:41.971960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.971943 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2" pod "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" (UID: "4770db10-c48e-4ac9-adb1-f2ed00d5e4e4"). InnerVolumeSpecName "kube-api-access-9dmz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:41.988389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:41.988348 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" podStartSLOduration=1.9883306809999999 podStartE2EDuration="1.988330681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:41.726847223 +0000 UTC m=+2256.867436167" watchObservedRunningTime="2026-11-01 05:40:41.988330681 +0000 UTC m=+2257.128919600" Nov 01 05:40:42.071497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.071438 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:42.071497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.071471 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:42.071728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.071508 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmz2\" (UniqueName: \"kubernetes.io/projected/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4-kube-api-access-9dmz2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:42.483681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.483591 4425 generic.go:356] "Generic (PLEG): container finished" podID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" containerID="f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583" exitCode=0 Nov 01 05:40:42.483681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.483644 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" Nov 01 05:40:42.484200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.483674 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" event={"ID":"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4","Type":"ContainerDied","Data":"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583"} Nov 01 05:40:42.484200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.483719 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9" event={"ID":"4770db10-c48e-4ac9-adb1-f2ed00d5e4e4","Type":"ContainerDied","Data":"d9033505552ce3416206bf9b0e947c69e6cc528cfe14bb56e2c384e63143d2ef"} Nov 01 05:40:42.484200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.483736 4425 scope.go:122] "RemoveContainer" containerID="f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583" Nov 01 05:40:42.494485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.494466 4425 scope.go:122] "RemoveContainer" containerID="f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583" Nov 01 05:40:42.494775 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:40:42.494751 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583\": container with ID starting with f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583 not found: ID does not exist" containerID="f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583" Nov 01 05:40:42.494841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.494784 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583"} err="failed to get container status \"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583\": rpc error: code = NotFound desc = could not find container \"f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583\": container with ID starting with f63b789da615386bed7456b6ff0238d83897dff253a4254e346cc7d355bc3583 not found: ID does not exist" Nov 01 05:40:42.652290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.652242 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:40:42.703082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.703040 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/auth-test-model-1c4191-kserve-76fff7877c-2w5q9"] Nov 01 05:40:42.974197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:42.974162 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" path="/var/lib/kubelet/pods/4770db10-c48e-4ac9-adb1-f2ed00d5e4e4/volumes" Nov 01 05:40:52.488118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:52.488086 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:40:52.488614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:52.488166 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:40:54.348477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.348443 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw"] Nov 01 05:40:54.349037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.349024 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" containerName="main" Nov 01 05:40:54.349093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.349038 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" containerName="main" Nov 01 05:40:54.349180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.349170 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4770db10-c48e-4ac9-adb1-f2ed00d5e4e4" containerName="main" Nov 01 05:40:54.353883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.353867 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.407822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.407784 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:40:54.412045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.412016 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw"] Nov 01 05:40:54.428114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.428075 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.428328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.428132 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.428398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.428338 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjqn\" (UniqueName: \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.458543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.458508 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:40:54.458818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.458800 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:40:54.458864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.458837 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:40:54.529682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.529646 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.529682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.529687 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.529942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.529774 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjqn\" (UniqueName: \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.530555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.530513 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.533178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.533152 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.592437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.592406 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjqn\" (UniqueName: \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\") pod \"maas-api-revoke-sub-30193dbb-qwdxw\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:54.668442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:54.668367 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:55.118445 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:55.118410 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd5024b_02b4_4e95_b2d7_16725bfb9ef8.slice/crio-3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c WatchSource:0}: Error finding container 3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c: Status 404 returned error can't find the container with id 3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c Nov 01 05:40:55.118686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.118657 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw"] Nov 01 05:40:55.578242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.578209 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" event={"ID":"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8","Type":"ContainerStarted","Data":"ef179d5ff308421e85c61b28a0f1883a1b4d3a854ceb55938932b90603acc443"} Nov 01 05:40:55.578703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.578293 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" event={"ID":"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8","Type":"ContainerStarted","Data":"3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c"} Nov 01 05:40:55.696569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.696521 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" podStartSLOduration=1.696506468 podStartE2EDuration="1.696506468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:55.696051614 +0000 UTC m=+2270.836640559" watchObservedRunningTime="2026-11-01 05:40:55.696506468 +0000 UTC m=+2270.837095387" Nov 01 05:40:55.977679 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.977649 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n"] Nov 01 05:40:55.980630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:55.980604 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4cbcb62b-qtd2n"] Nov 01 05:40:56.586458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:56.586423 4425 generic.go:356] "Generic (PLEG): container finished" podID="7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" containerID="ef179d5ff308421e85c61b28a0f1883a1b4d3a854ceb55938932b90603acc443" exitCode=0 Nov 01 05:40:56.586863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:56.586511 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" event={"ID":"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8","Type":"ContainerDied","Data":"ef179d5ff308421e85c61b28a0f1883a1b4d3a854ceb55938932b90603acc443"} Nov 01 05:40:56.949293 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:40:56.949195 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:40:56.975764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:56.975730 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a290ea0-1c09-4fb2-9edc-b1e73a1f985f" path="/var/lib/kubelet/pods/7a290ea0-1c09-4fb2-9edc-b1e73a1f985f/volumes" Nov 01 05:40:57.788576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.788545 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:57.793584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.793564 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\" (UniqueName: \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\") pod \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " Nov 01 05:40:57.793706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.793654 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\") pod \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " Nov 01 05:40:57.793706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.793694 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\") pod \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\" (UID: \"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8\") " Nov 01 05:40:57.794040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.794016 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca" pod "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" (UID: "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:40:57.795804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.795782 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca" pod "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" (UID: "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:40:57.795904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.795835 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn" pod "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" (UID: "7cd5024b-02b4-4e95-b2d7-16725bfb9ef8"). InnerVolumeSpecName "kube-api-access-vfjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:40:57.895704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.895619 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:57.895704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.895649 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:57.895704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:57.895659 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjqn\" (UniqueName: \"kubernetes.io/projected/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8-kube-api-access-vfjqn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:40:58.595911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.595877 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:40:58.596630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.596609 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" containerName="revoke-keys" Nov 01 05:40:58.596630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.596628 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" containerName="revoke-keys" Nov 01 05:40:58.596721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.596710 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" containerName="revoke-keys" Nov 01 05:40:58.600322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.600301 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.625660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.625639 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" Nov 01 05:40:58.625660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.625653 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw" event={"ID":"7cd5024b-02b4-4e95-b2d7-16725bfb9ef8","Type":"ContainerDied","Data":"3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c"} Nov 01 05:40:58.625831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.625701 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0dbb3900d98207825efb678c331f5a08efa70944a005e63e3fab4f534f664c" Nov 01 05:40:58.654204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.654166 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad90c52d-openshift-default-dockercfg-bb25b\"" Nov 01 05:40:58.704596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704440 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704482 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704572 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704628 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhwp\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704754 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704854 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.704891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.704884 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.705215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.705023 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.705215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.705121 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.710423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.710396 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:40:58.806714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806677 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806732 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806767 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806788 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806820 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806854 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhwp\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.806953 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807051 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807085 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807205 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807301 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807359 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807540 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.807800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.807659 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.809391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.809370 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.809504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.809442 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.885329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.885071 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhwp\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.905315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.905239 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\") pod \"e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:58.924562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:58.924525 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:40:59.365208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.365181 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:40:59.365789 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:40:59.365768 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c211a5_586a_4f41_91b7_738cae284326.slice/crio-dde30f2c5283fbfffbc12e63f0c4b621c2f16268ffbde22063cb64b8a1ff6cc3 WatchSource:0}: Error finding container dde30f2c5283fbfffbc12e63f0c4b621c2f16268ffbde22063cb64b8a1ff6cc3: Status 404 returned error can't find the container with id dde30f2c5283fbfffbc12e63f0c4b621c2f16268ffbde22063cb64b8a1ff6cc3 Nov 01 05:40:59.368426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.368394 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:59.368498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.368462 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:59.368534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.368498 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:40:59.635885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.635798 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerStarted","Data":"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3"} Nov 01 05:40:59.635885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.635855 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerStarted","Data":"dde30f2c5283fbfffbc12e63f0c4b621c2f16268ffbde22063cb64b8a1ff6cc3"} Nov 01 05:40:59.760954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.760881 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podStartSLOduration=1.7608585030000001 podStartE2EDuration="1.760858503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:40:59.757372699 +0000 UTC m=+2274.897961638" watchObservedRunningTime="2026-11-01 05:40:59.760858503 +0000 UTC m=+2274.901447420" Nov 01 05:40:59.925278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:40:59.925158 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:41:00.932525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:00.932487 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:41:01.650785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:01.650748 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:41:01.651806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:01.651782 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:41:06.157453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.157422 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb"] Nov 01 05:41:06.161943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.161923 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.215269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.215222 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:41:06.216919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.216892 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td56w\" (UniqueName: \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.217078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.216945 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.217149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.217072 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.219240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.219216 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb"] Nov 01 05:41:06.268502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.268467 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:41:06.268729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.268565 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:41:06.268729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.268647 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:41:06.318332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.318294 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td56w\" (UniqueName: \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.318533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.318361 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.318533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.318400 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.319121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.319093 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.321002 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.320982 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.380992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.380964 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td56w\" (UniqueName: \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\") pod \"maas-api-revoke-sub-77965773-sklzb\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.476990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.476906 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:06.751476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.751396 4425 scope.go:122] "RemoveContainer" containerID="decf6b535fe92defe8e9d5afd5079a30bf98328a4564a62cffacd90d0ffadb26" Nov 01 05:41:06.855585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:06.855556 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb"] Nov 01 05:41:06.856104 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:41:06.856076 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6387c9a_c3d7_4374_bb07_f3bac5e2387e.slice/crio-24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2 WatchSource:0}: Error finding container 24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2: Status 404 returned error can't find the container with id 24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2 Nov 01 05:41:07.717972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:07.717936 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" event={"ID":"f6387c9a-c3d7-4374-bb07-f3bac5e2387e","Type":"ContainerStarted","Data":"1464d75cefebff47f9fcec3eb873d42bea6f474ef24057cfe86f56b2a66d24c8"} Nov 01 05:41:07.718361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:07.717996 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" event={"ID":"f6387c9a-c3d7-4374-bb07-f3bac5e2387e","Type":"ContainerStarted","Data":"24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2"} Nov 01 05:41:07.837764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:07.837714 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" podStartSLOduration=1.837700759 podStartE2EDuration="1.837700759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:07.835870269 +0000 UTC m=+2282.976459190" watchObservedRunningTime="2026-11-01 05:41:07.837700759 +0000 UTC m=+2282.978289721" Nov 01 05:41:08.726450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:08.726413 4425 generic.go:356] "Generic (PLEG): container finished" podID="f6387c9a-c3d7-4374-bb07-f3bac5e2387e" containerID="1464d75cefebff47f9fcec3eb873d42bea6f474ef24057cfe86f56b2a66d24c8" exitCode=0 Nov 01 05:41:08.726859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:08.726501 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" event={"ID":"f6387c9a-c3d7-4374-bb07-f3bac5e2387e","Type":"ContainerDied","Data":"1464d75cefebff47f9fcec3eb873d42bea6f474ef24057cfe86f56b2a66d24c8"} Nov 01 05:41:09.893789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.893763 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:09.964725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.964684 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\") pod \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " Nov 01 05:41:09.964913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.964790 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\") pod \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " Nov 01 05:41:09.964960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.964944 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\" (UniqueName: \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\") pod \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\" (UID: \"f6387c9a-c3d7-4374-bb07-f3bac5e2387e\") " Nov 01 05:41:09.965221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.965197 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca" pod "f6387c9a-c3d7-4374-bb07-f3bac5e2387e" (UID: "f6387c9a-c3d7-4374-bb07-f3bac5e2387e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:09.965484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.965463 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:09.967127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.967095 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca" pod "f6387c9a-c3d7-4374-bb07-f3bac5e2387e" (UID: "f6387c9a-c3d7-4374-bb07-f3bac5e2387e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:41:09.967234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:09.967162 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w" pod "f6387c9a-c3d7-4374-bb07-f3bac5e2387e" (UID: "f6387c9a-c3d7-4374-bb07-f3bac5e2387e"). InnerVolumeSpecName "kube-api-access-td56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:10.067363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:10.067327 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td56w\" (UniqueName: \"kubernetes.io/projected/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-kube-api-access-td56w\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:10.067363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:10.067359 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6387c9a-c3d7-4374-bb07-f3bac5e2387e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:10.744132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:10.744090 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" event={"ID":"f6387c9a-c3d7-4374-bb07-f3bac5e2387e","Type":"ContainerDied","Data":"24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2"} Nov 01 05:41:10.744132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:10.744117 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb" Nov 01 05:41:10.744455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:10.744185 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ddf9631ad4d7bb9a6fb80f40e2d26e24e902e50843b84c610fb6d5d5fdc7c2" Nov 01 05:41:25.763404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.763174 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2"] Nov 01 05:41:25.766024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.765827 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f6387c9a-c3d7-4374-bb07-f3bac5e2387e" containerName="revoke-keys" Nov 01 05:41:25.766024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.765857 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6387c9a-c3d7-4374-bb07-f3bac5e2387e" containerName="revoke-keys" Nov 01 05:41:25.766351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.766067 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6387c9a-c3d7-4374-bb07-f3bac5e2387e" containerName="revoke-keys" Nov 01 05:41:25.771784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.771757 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:25.822780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.822752 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2"] Nov 01 05:41:25.869574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.869506 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:41:25.869574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.869561 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:41:25.869574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.869550 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:41:25.869818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.869535 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:41:25.909431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.909397 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:25.909613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.909482 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:25.909613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:25.909580 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmmv\" (UniqueName: \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.010361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.010326 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmmv\" (UniqueName: \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.010538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.010475 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.010590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.010536 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.011233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.011204 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.013554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.013497 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.072296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.072237 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmmv\" (UniqueName: \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\") pod \"maas-api-revoke-sub-7f80f5ce-gkmf2\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.089901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.089868 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:26.471057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.471027 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2"] Nov 01 05:41:26.471580 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:41:26.471548 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cd8998_782e_4942_8767_83e5c019c319.slice/crio-50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc WatchSource:0}: Error finding container 50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc: Status 404 returned error can't find the container with id 50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc Nov 01 05:41:26.860862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.860826 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" event={"ID":"35cd8998-782e-4942-8767-83e5c019c319","Type":"ContainerStarted","Data":"fc2de4cd15dc97c13113b574a5a7bd62b1ecd23f039f0e43d8cd60003e93b164"} Nov 01 05:41:26.861323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.860890 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" event={"ID":"35cd8998-782e-4942-8767-83e5c019c319","Type":"ContainerStarted","Data":"50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc"} Nov 01 05:41:26.947898 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:26.947857 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:41:26.980742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:26.980693 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" podStartSLOduration=1.98067743 podStartE2EDuration="1.98067743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:26.978748298 +0000 UTC m=+2302.119337241" watchObservedRunningTime="2026-11-01 05:41:26.98067743 +0000 UTC m=+2302.121266365" Nov 01 05:41:27.875188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:27.875139 4425 generic.go:356] "Generic (PLEG): container finished" podID="35cd8998-782e-4942-8767-83e5c019c319" containerID="fc2de4cd15dc97c13113b574a5a7bd62b1ecd23f039f0e43d8cd60003e93b164" exitCode=0 Nov 01 05:41:27.875725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:27.875287 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" event={"ID":"35cd8998-782e-4942-8767-83e5c019c319","Type":"ContainerDied","Data":"fc2de4cd15dc97c13113b574a5a7bd62b1ecd23f039f0e43d8cd60003e93b164"} Nov 01 05:41:29.041723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.041697 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:29.156054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.155965 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\" (UniqueName: \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\") pod \"35cd8998-782e-4942-8767-83e5c019c319\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " Nov 01 05:41:29.156054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.156034 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\") pod \"35cd8998-782e-4942-8767-83e5c019c319\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " Nov 01 05:41:29.156343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.156168 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\") pod \"35cd8998-782e-4942-8767-83e5c019c319\" (UID: \"35cd8998-782e-4942-8767-83e5c019c319\") " Nov 01 05:41:29.156607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.156573 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca" pod "35cd8998-782e-4942-8767-83e5c019c319" (UID: "35cd8998-782e-4942-8767-83e5c019c319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:29.158758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.158732 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca" pod "35cd8998-782e-4942-8767-83e5c019c319" (UID: "35cd8998-782e-4942-8767-83e5c019c319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:41:29.158930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.158847 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv" pod "35cd8998-782e-4942-8767-83e5c019c319" (UID: "35cd8998-782e-4942-8767-83e5c019c319"). InnerVolumeSpecName "kube-api-access-qxmmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:29.258296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.258237 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35cd8998-782e-4942-8767-83e5c019c319-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:29.258296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.258291 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35cd8998-782e-4942-8767-83e5c019c319-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:29.258296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.258305 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxmmv\" (UniqueName: \"kubernetes.io/projected/35cd8998-782e-4942-8767-83e5c019c319-kube-api-access-qxmmv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:29.889856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.889819 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" Nov 01 05:41:29.890072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.889855 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2" event={"ID":"35cd8998-782e-4942-8767-83e5c019c319","Type":"ContainerDied","Data":"50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc"} Nov 01 05:41:29.890072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:29.889908 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c2f32a764f2443b9331bc3097b662f1d7931e0adb8caa57b7136df4777d3bc" Nov 01 05:41:30.283068 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.282991 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:30.283602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.283587 4425 cpu_manager.go:414] "containerMap: removing container" podUID="35cd8998-782e-4942-8767-83e5c019c319" containerName="revoke-keys" Nov 01 05:41:30.283651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.283604 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cd8998-782e-4942-8767-83e5c019c319" containerName="revoke-keys" Nov 01 05:41:30.283751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.283742 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cd8998-782e-4942-8767-83e5c019c319" containerName="revoke-keys" Nov 01 05:41:30.287226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.287205 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.362914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.362880 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:30.474904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.474867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnhm\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.475079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.474910 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.475122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.475076 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.475171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.475122 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.576901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.576861 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnhm\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.576901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.576904 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.577166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.576959 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.577166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.576975 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.577692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.577661 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.577810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.577766 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.579790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.579767 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.639372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.639334 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnhm\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:30.901777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:30.901687 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:31.286715 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:41:31.286687 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a2c7f9_f8c4_40a1_beca_419f75242b48.slice/crio-a522f8d2b58cc12b7281963ab0bc17de0b9eead396a92048890182125e96845f WatchSource:0}: Error finding container a522f8d2b58cc12b7281963ab0bc17de0b9eead396a92048890182125e96845f: Status 404 returned error can't find the container with id a522f8d2b58cc12b7281963ab0bc17de0b9eead396a92048890182125e96845f Nov 01 05:41:31.287092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:31.287011 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:31.904498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:31.904455 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" event={"ID":"b9a2c7f9-f8c4-40a1-beca-419f75242b48","Type":"ContainerStarted","Data":"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50"} Nov 01 05:41:31.904696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:31.904517 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" event={"ID":"b9a2c7f9-f8c4-40a1-beca-419f75242b48","Type":"ContainerStarted","Data":"a522f8d2b58cc12b7281963ab0bc17de0b9eead396a92048890182125e96845f"} Nov 01 05:41:31.904696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:31.904668 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:32.026874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:32.026822 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" podStartSLOduration=2.026807149 podStartE2EDuration="2.026807149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:32.023801991 +0000 UTC m=+2307.164390974" watchObservedRunningTime="2026-11-01 05:41:32.026807149 +0000 UTC m=+2307.167396067" Nov 01 05:41:37.299864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.299752 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z"] Nov 01 05:41:37.305988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.305070 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.363406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.362468 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.363406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.362571 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qzv\" (UniqueName: \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.363406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.362640 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.369239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.369189 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z"] Nov 01 05:41:37.407366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.407329 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:41:37.407366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.407352 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:41:37.407366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.407368 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:41:37.407689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.407466 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:41:37.464495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.464450 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.464717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.464541 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qzv\" (UniqueName: \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.464717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.464594 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.465445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.465407 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.467144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.467122 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.526651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.526613 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qzv\" (UniqueName: \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\") pod \"maas-api-revoke-sub-ab085c4a-hfp5z\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:37.620119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:37.620069 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:38.001155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:38.001124 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z"] Nov 01 05:41:38.001882 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:41:38.001858 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4f4df5_0fa1_4810_b6de_a59746093180.slice/crio-cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef WatchSource:0}: Error finding container cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef: Status 404 returned error can't find the container with id cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef Nov 01 05:41:38.954960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:38.954930 4425 generic.go:356] "Generic (PLEG): container finished" podID="8e4f4df5-0fa1-4810-b6de-a59746093180" containerID="3796e320ce4ee0bca94a62685cfd5aa3add44f9e0cf93bba026a2720f2ac025d" exitCode=0 Nov 01 05:41:38.955396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:38.955012 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" event={"ID":"8e4f4df5-0fa1-4810-b6de-a59746093180","Type":"ContainerDied","Data":"3796e320ce4ee0bca94a62685cfd5aa3add44f9e0cf93bba026a2720f2ac025d"} Nov 01 05:41:38.955396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:38.955064 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" event={"ID":"8e4f4df5-0fa1-4810-b6de-a59746093180","Type":"ContainerStarted","Data":"cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef"} Nov 01 05:41:40.109763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.109733 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:40.201928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.201886 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\") pod \"8e4f4df5-0fa1-4810-b6de-a59746093180\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " Nov 01 05:41:40.202132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.202014 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\") pod \"8e4f4df5-0fa1-4810-b6de-a59746093180\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " Nov 01 05:41:40.202204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.202139 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\" (UniqueName: \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\") pod \"8e4f4df5-0fa1-4810-b6de-a59746093180\" (UID: \"8e4f4df5-0fa1-4810-b6de-a59746093180\") " Nov 01 05:41:40.202306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.202282 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca" pod "8e4f4df5-0fa1-4810-b6de-a59746093180" (UID: "8e4f4df5-0fa1-4810-b6de-a59746093180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:40.202669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.202648 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:40.204392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.204367 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca" pod "8e4f4df5-0fa1-4810-b6de-a59746093180" (UID: "8e4f4df5-0fa1-4810-b6de-a59746093180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:41:40.204542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.204521 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv" pod "8e4f4df5-0fa1-4810-b6de-a59746093180" (UID: "8e4f4df5-0fa1-4810-b6de-a59746093180"). InnerVolumeSpecName "kube-api-access-d4qzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:40.303667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.303624 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4f4df5-0fa1-4810-b6de-a59746093180-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:40.303667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.303659 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qzv\" (UniqueName: \"kubernetes.io/projected/8e4f4df5-0fa1-4810-b6de-a59746093180-kube-api-access-d4qzv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:40.970726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.970692 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" Nov 01 05:41:40.975737 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.975706 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z" event={"ID":"8e4f4df5-0fa1-4810-b6de-a59746093180","Type":"ContainerDied","Data":"cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef"} Nov 01 05:41:40.975910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:40.975753 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf6fba798453154d09a90ad36325754d0770fb2e02b7b9ee3244b0ab9c49cbef" Nov 01 05:41:42.917568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:42.917518 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:41:51.037813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.037778 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw"] Nov 01 05:41:51.038659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.038638 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8e4f4df5-0fa1-4810-b6de-a59746093180" containerName="revoke-keys" Nov 01 05:41:51.038720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.038663 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4f4df5-0fa1-4810-b6de-a59746093180" containerName="revoke-keys" Nov 01 05:41:51.038827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.038815 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4f4df5-0fa1-4810-b6de-a59746093180" containerName="revoke-keys" Nov 01 05:41:51.043963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.043935 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.099636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.099601 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw"] Nov 01 05:41:51.145678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.145642 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:41:51.145678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.145643 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:41:51.145678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.145648 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:41:51.147940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.147917 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:41:51.153652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.153624 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsnn\" (UniqueName: \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.153801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.153778 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.153928 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.153909 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.191626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.191591 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:41:51.196021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.195998 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.254205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.254169 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:41:51.254758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.254732 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsnn\" (UniqueName: \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.254879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.254786 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.254879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.254817 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.255030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.255001 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.255085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.255071 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4np\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.255138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.255111 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.255291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.255237 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.255834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.255808 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.261410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.261385 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.352154 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsnn\" (UniqueName: \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\") pod \"maas-api-revoke-sub-aa80308f-lwrbw\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.357123 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4np\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.357181 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.357412 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.357442 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.358280 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.358619 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:51.362316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.359132 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.386838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.385475 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.421976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.421937 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4np\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.522277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.517167 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:51.830475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:51.830438 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw"] Nov 01 05:41:52.064387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:52.064031 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:41:52.077157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:52.077126 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" event={"ID":"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf","Type":"ContainerStarted","Data":"53a40a729b1037c9529f5b1c32af07dd4d970ccdfa17a2a7b4700665031f8ef9"} Nov 01 05:41:52.099860 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.099743 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:41:52.099860 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.099771 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:41:52.099860 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.099781 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:41:52.099860 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.099795 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:52.100232 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.099865 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:41:52.599837066 +0000 UTC m=+2327.740425967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:52.605874 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.605833 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:41:52.605874 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.605866 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:41:52.605874 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.605878 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:41:52.606116 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.605888 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:52.606116 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:52.605951 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:41:53.605936077 +0000 UTC m=+2328.746524992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:52.914651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:52.914615 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:41:53.085692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.085648 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" event={"ID":"0d53569d-5233-4b9c-a20f-509e3ff68b13","Type":"ContainerStarted","Data":"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792"} Nov 01 05:41:53.086182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.085718 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" event={"ID":"0d53569d-5233-4b9c-a20f-509e3ff68b13","Type":"ContainerStarted","Data":"a4ee251d83933822afa037060cc392f3d9199053f6a9d65e65a47a706713d885"} Nov 01 05:41:53.086182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.085745 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:41:53.087454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.087430 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" event={"ID":"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf","Type":"ContainerStarted","Data":"65bb6ec57612583fd60a18e6c413ca93a213fbec939dec13928f51b70565b9f0"} Nov 01 05:41:53.205168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.205111 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" podStartSLOduration=2.205095409 podStartE2EDuration="2.205095409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:53.203752813 +0000 UTC m=+2328.344341735" watchObservedRunningTime="2026-11-01 05:41:53.205095409 +0000 UTC m=+2328.345684373" Nov 01 05:41:53.328241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.328185 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" podStartSLOduration=3.3281718270000002 podStartE2EDuration="3.328171827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:53.324984058 +0000 UTC m=+2328.465573002" watchObservedRunningTime="2026-11-01 05:41:53.328171827 +0000 UTC m=+2328.468760774" Nov 01 05:41:53.619836 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:53.619805 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:41:53.619836 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:53.619831 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:41:53.619836 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:53.619842 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:41:53.620094 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:53.619851 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:53.620094 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:53.619907 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:41:55.619891093 +0000 UTC m=+2330.760480007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:53.834161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.834125 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:53.835524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:53.835493 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:41:54.095814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:54.095778 4425 generic.go:356] "Generic (PLEG): container finished" podID="6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" containerID="65bb6ec57612583fd60a18e6c413ca93a213fbec939dec13928f51b70565b9f0" exitCode=0 Nov 01 05:41:54.096308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:54.095839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" event={"ID":"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf","Type":"ContainerDied","Data":"65bb6ec57612583fd60a18e6c413ca93a213fbec939dec13928f51b70565b9f0"} Nov 01 05:41:55.312907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.312881 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:55.448441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.446790 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\" (UniqueName: \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\") pod \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " Nov 01 05:41:55.448441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.446867 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\") pod \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " Nov 01 05:41:55.448441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.446915 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\") pod \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\" (UID: \"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf\") " Nov 01 05:41:55.468284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.455371 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca" pod "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" (UID: "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:55.468284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.455723 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:55.471517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.471474 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn" pod "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" (UID: "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf"). InnerVolumeSpecName "kube-api-access-ttsnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:55.474887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.474849 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca" pod "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" (UID: "6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:41:55.563443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.563381 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttsnn\" (UniqueName: \"kubernetes.io/projected/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-kube-api-access-ttsnn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:55.566400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:55.566368 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:55.669065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:55.668388 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:41:55.669065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:55.668418 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:41:55.669065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:55.668434 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:41:55.669065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:55.668446 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:55.669065 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:55.668508 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:41:59.668488843 +0000 UTC m=+2334.809077757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:56.110401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:56.110368 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" Nov 01 05:41:56.110401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:56.110397 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw" event={"ID":"6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf","Type":"ContainerDied","Data":"53a40a729b1037c9529f5b1c32af07dd4d970ccdfa17a2a7b4700665031f8ef9"} Nov 01 05:41:56.110626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:56.110441 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a40a729b1037c9529f5b1c32af07dd4d970ccdfa17a2a7b4700665031f8ef9" Nov 01 05:41:56.424180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:56.424079 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:41:56.424697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:56.424383 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" containerID="cri-o://c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7" gracePeriod=30 Nov 01 05:41:56.970531 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:56.970488 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:41:58.835429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:58.835386 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" containerID="cri-o://021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50" gracePeriod=25 Nov 01 05:41:59.116092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.116058 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:59.135065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.135022 4425 generic.go:356] "Generic (PLEG): container finished" podID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerID="021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50" exitCode=0 Nov 01 05:41:59.135311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.135082 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" Nov 01 05:41:59.135311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.135119 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" event={"ID":"b9a2c7f9-f8c4-40a1-beca-419f75242b48","Type":"ContainerDied","Data":"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50"} Nov 01 05:41:59.135311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.135189 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg" event={"ID":"b9a2c7f9-f8c4-40a1-beca-419f75242b48","Type":"ContainerDied","Data":"a522f8d2b58cc12b7281963ab0bc17de0b9eead396a92048890182125e96845f"} Nov 01 05:41:59.135311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.135204 4425 scope.go:122] "RemoveContainer" containerID="021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50" Nov 01 05:41:59.146281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.146232 4425 scope.go:122] "RemoveContainer" containerID="021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50" Nov 01 05:41:59.146673 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.146638 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50\": container with ID starting with 021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50 not found: ID does not exist" containerID="021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50" Nov 01 05:41:59.146799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.146683 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50"} err="failed to get container status \"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50\": rpc error: code = NotFound desc = could not find container \"021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50\": container with ID starting with 021946a5f4f00a696a825b3fe64d4055e1a7abf274c1e4c7bc76d9258b136e50 not found: ID does not exist" Nov 01 05:41:59.217037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.217003 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\") pod \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " Nov 01 05:41:59.217235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.217153 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\") pod \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " Nov 01 05:41:59.217333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.217291 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\") pod \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " Nov 01 05:41:59.217462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.217441 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\") pod \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\" (UID: \"b9a2c7f9-f8c4-40a1-beca-419f75242b48\") " Nov 01 05:41:59.218403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.218103 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume" pod "b9a2c7f9-f8c4-40a1-beca-419f75242b48" (UID: "b9a2c7f9-f8c4-40a1-beca-419f75242b48"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:59.218403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.218300 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:59.218688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.218665 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume" pod "b9a2c7f9-f8c4-40a1-beca-419f75242b48" (UID: "b9a2c7f9-f8c4-40a1-beca-419f75242b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:41:59.219826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.219796 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm" pod "b9a2c7f9-f8c4-40a1-beca-419f75242b48" (UID: "b9a2c7f9-f8c4-40a1-beca-419f75242b48"). InnerVolumeSpecName "kube-api-access-ffnhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:59.221069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.221033 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials" pod "b9a2c7f9-f8c4-40a1-beca-419f75242b48" (UID: "b9a2c7f9-f8c4-40a1-beca-419f75242b48"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:41:59.319371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.319334 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:59.319371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.319363 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a2c7f9-f8c4-40a1-beca-419f75242b48-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:59.319371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.319373 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffnhm\" (UniqueName: \"kubernetes.io/projected/b9a2c7f9-f8c4-40a1-beca-419f75242b48-kube-api-access-ffnhm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:41:59.559495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.559452 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:59.562954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:41:59.562927 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-7bcbc9bfg"] Nov 01 05:41:59.725409 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.725325 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:41:59.725409 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.725354 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:41:59.725409 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.725365 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:41:59.725409 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.725375 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:41:59.725686 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:41:59.725441 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:42:07.725425725 +0000 UTC m=+2342.866014622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:42:00.233099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:00.233065 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:42:00.233527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:00.233138 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:42:00.975869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:00.975827 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" path="/var/lib/kubelet/pods/b9a2c7f9-f8c4-40a1-beca-419f75242b48/volumes" Nov 01 05:42:01.694747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.694719 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:42:01.854172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854116 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854216 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854246 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854287 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854334 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854366 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854431 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854564 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854645 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\") pod \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\" (UID: \"9785f724-7bbc-4a8f-b065-1b3a23ac8a79\") " Nov 01 05:42:01.854938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.854910 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:42:01.855339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.855231 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.855339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.855267 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:42:01.855339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.855335 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:42:01.855568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.855508 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:42:01.855729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.855707 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:01.857328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.857301 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "kube-api-access-2cxv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:01.857657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.857628 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:01.857763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.857672 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:42:01.857827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.857786 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy" pod "9785f724-7bbc-4a8f-b065-1b3a23ac8a79" (UID: "9785f724-7bbc-4a8f-b065-1b3a23ac8a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:42:01.956553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956507 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956543 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956556 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956569 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxv2\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-kube-api-access-2cxv2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956582 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956594 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956605 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:01.956813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:01.956617 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9785f724-7bbc-4a8f-b065-1b3a23ac8a79-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:02.167494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.167401 4425 generic.go:356] "Generic (PLEG): container finished" podID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerID="6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2" exitCode=0 Nov 01 05:42:02.167494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.167472 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerDied","Data":"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2"} Nov 01 05:42:02.167927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.167909 4425 scope.go:122] "RemoveContainer" containerID="6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2" Nov 01 05:42:02.169095 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169069 4425 generic.go:356] "Generic (PLEG): container finished" podID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerID="c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7" exitCode=0 Nov 01 05:42:02.169211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169121 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" Nov 01 05:42:02.169211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169150 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" event={"ID":"9785f724-7bbc-4a8f-b065-1b3a23ac8a79","Type":"ContainerDied","Data":"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7"} Nov 01 05:42:02.169373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169219 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t" event={"ID":"9785f724-7bbc-4a8f-b065-1b3a23ac8a79","Type":"ContainerDied","Data":"50b6a7d4bacc5588f4a1f4069742993ab65836d88e5d767b1f0f1fe66335514e"} Nov 01 05:42:02.169373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169285 4425 scope.go:122] "RemoveContainer" containerID="c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7" Nov 01 05:42:02.169722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169697 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:02.169788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169774 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:02.169837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.169821 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:02.184026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.184004 4425 scope.go:122] "RemoveContainer" containerID="c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7" Nov 01 05:42:02.184405 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.184382 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7\": container with ID starting with c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7 not found: ID does not exist" containerID="c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7" Nov 01 05:42:02.184482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.184417 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7"} err="failed to get container status \"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7\": rpc error: code = NotFound desc = could not find container \"c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7\": container with ID starting with c821c2a7e0aa2079387941af14ba4fc7c4f639eb15a9a80b0df6e3c05387dac7 not found: ID does not exist" Nov 01 05:42:02.406354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.406321 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:42:02.411865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.411837 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-n8g9t"] Nov 01 05:42:02.653817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.653780 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq"] Nov 01 05:42:02.654505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654489 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" containerName="revoke-keys" Nov 01 05:42:02.654564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654507 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" containerName="revoke-keys" Nov 01 05:42:02.654564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654523 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" Nov 01 05:42:02.654564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654529 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" Nov 01 05:42:02.654564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654539 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" Nov 01 05:42:02.654564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654545 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" Nov 01 05:42:02.654716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654652 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" containerName="revoke-keys" Nov 01 05:42:02.654716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654662 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" containerName="istio-proxy" Nov 01 05:42:02.654716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.654668 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a2c7f9-f8c4-40a1-beca-419f75242b48" containerName="payload-processing" Nov 01 05:42:02.668289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.668229 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.716018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.715980 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq"] Nov 01 05:42:02.722442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.722391 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:42:02.722442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.722420 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:42:02.722664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.722488 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:42:02.722664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.722520 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:42:02.778281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.776576 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.778281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.776688 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6qh\" (UniqueName: \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.778281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.776748 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.877656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.877613 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6qh\" (UniqueName: \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.877851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.877679 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.877851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.877801 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.878491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.878462 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.880734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.880706 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.940455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.940378 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6qh\" (UniqueName: \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\") pod \"maas-api-revoke-sub-aea1b325-zbfsq\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:02.976321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.976287 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9785f724-7bbc-4a8f-b065-1b3a23ac8a79" path="/var/lib/kubelet/pods/9785f724-7bbc-4a8f-b065-1b3a23ac8a79/volumes" Nov 01 05:42:02.978762 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.978732 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:02.978762 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.978761 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:02.978969 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.978771 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:02.978969 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.978783 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:02.978969 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:02.978873 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:03.478850474 +0000 UTC m=+2338.619439384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:02.983225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:02.983202 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:03.179781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:03.179747 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerStarted","Data":"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf"} Nov 01 05:42:03.419629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:03.419596 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq"] Nov 01 05:42:03.419799 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:42:03.419779 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd945ff35_d6e1_400c_bc12_89f7175e9abc.slice/crio-b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c WatchSource:0}: Error finding container b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c: Status 404 returned error can't find the container with id b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c Nov 01 05:42:03.487927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:03.487894 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:03.487927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:03.487921 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:03.487927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:03.487928 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:03.488145 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:03.487938 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:03.488145 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:03.488003 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:04.48798784 +0000 UTC m=+2339.628576752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:03.706397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:03.706363 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:42:03.707917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:03.707889 4425 patch_prober.go:29] interesting pod/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 05:42:03.708063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:03.707955 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 05:42:04.099109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.099071 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:42:04.189734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.189696 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" event={"ID":"d945ff35-d6e1-400c-bc12-89f7175e9abc","Type":"ContainerStarted","Data":"63de68ea7241550e97b429e83a4e00e6b9890dc5a372bf4aca6707f4b0fa17ec"} Nov 01 05:42:04.189964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.189764 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" event={"ID":"d945ff35-d6e1-400c-bc12-89f7175e9abc","Type":"ContainerStarted","Data":"b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c"} Nov 01 05:42:04.328308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.328225 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" podStartSLOduration=2.328205868 podStartE2EDuration="2.328205868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:42:04.322795014 +0000 UTC m=+2339.463383954" watchObservedRunningTime="2026-11-01 05:42:04.328205868 +0000 UTC m=+2339.468794788" Nov 01 05:42:04.501422 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:04.501338 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:04.501422 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:04.501367 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:04.501422 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:04.501377 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:04.501422 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:04.501386 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:04.501758 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:04.501446 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:06.501430349 +0000 UTC m=+2341.642019255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:04.706084 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.706053 4425 patch_prober.go:29] interesting pod/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 05:42:04.706286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:04.706111 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 05:42:05.068413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.068373 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb"] Nov 01 05:42:05.072201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.072167 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b9ff7c-zwgcb"] Nov 01 05:42:05.198535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.198495 4425 generic.go:356] "Generic (PLEG): container finished" podID="d945ff35-d6e1-400c-bc12-89f7175e9abc" containerID="63de68ea7241550e97b429e83a4e00e6b9890dc5a372bf4aca6707f4b0fa17ec" exitCode=0 Nov 01 05:42:05.198941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.198576 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" event={"ID":"d945ff35-d6e1-400c-bc12-89f7175e9abc","Type":"ContainerDied","Data":"63de68ea7241550e97b429e83a4e00e6b9890dc5a372bf4aca6707f4b0fa17ec"} Nov 01 05:42:05.431456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.431375 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:42:05.706622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.706529 4425 patch_prober.go:29] interesting pod/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 05:42:05.706622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:05.706600 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 05:42:06.402951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.402922 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:06.426515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.426488 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\") pod \"d945ff35-d6e1-400c-bc12-89f7175e9abc\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " Nov 01 05:42:06.426655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.426541 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\" (UniqueName: \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\") pod \"d945ff35-d6e1-400c-bc12-89f7175e9abc\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " Nov 01 05:42:06.426655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.426566 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\") pod \"d945ff35-d6e1-400c-bc12-89f7175e9abc\" (UID: \"d945ff35-d6e1-400c-bc12-89f7175e9abc\") " Nov 01 05:42:06.427031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.427009 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca" pod "d945ff35-d6e1-400c-bc12-89f7175e9abc" (UID: "d945ff35-d6e1-400c-bc12-89f7175e9abc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:06.428740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.428718 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca" pod "d945ff35-d6e1-400c-bc12-89f7175e9abc" (UID: "d945ff35-d6e1-400c-bc12-89f7175e9abc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:42:06.428834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.428808 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh" pod "d945ff35-d6e1-400c-bc12-89f7175e9abc" (UID: "d945ff35-d6e1-400c-bc12-89f7175e9abc"). InnerVolumeSpecName "kube-api-access-4s6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.529059 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.529095 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6qh\" (UniqueName: \"kubernetes.io/projected/d945ff35-d6e1-400c-bc12-89f7175e9abc-kube-api-access-4s6qh\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.529110 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d945ff35-d6e1-400c-bc12-89f7175e9abc-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:06.529170 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:06.529193 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:06.529201 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:06.529214 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:06.530784 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:06.529317 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:10.529293742 +0000 UTC m=+2345.669882653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:06.706422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.706385 4425 patch_prober.go:29] interesting pod/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 05:42:06.706586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.706441 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 05:42:06.793860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.793779 4425 scope.go:122] "RemoveContainer" containerID="9f471d47c35984eb8d44d73b2f137f567a069fba87ca3867e3f13fe6fb8bd168" Nov 01 05:42:06.975547 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:06.975509 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f" path="/var/lib/kubelet/pods/c2b4be78-0bf2-4641-a6a1-d1bc41b3a95f/volumes" Nov 01 05:42:07.216492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:07.216467 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" Nov 01 05:42:07.216660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:07.216510 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq" event={"ID":"d945ff35-d6e1-400c-bc12-89f7175e9abc","Type":"ContainerDied","Data":"b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c"} Nov 01 05:42:07.216660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:07.216561 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33a560d5d736bcd684d0ebd5235ea1622b58fd41202340b4c8572f7ff6a5f3c" Nov 01 05:42:07.706262 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:07.706229 4425 patch_prober.go:29] interesting pod/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 05:42:07.706732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:07.706311 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 05:42:07.765278 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:07.764917 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-2e04ffa38ad1: secret "agc-cred-2e04ffa38ad1" not found Nov 01 05:42:07.765278 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:07.764952 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-388566305a92: secret "agc-cred-388566305a92" not found Nov 01 05:42:07.765278 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:07.764966 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-500efa28b76e: secret "agc-cred-500efa28b76e" not found Nov 01 05:42:07.765278 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:07.764979 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf: [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:42:07.765278 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:07.765048 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials podName:7f496e24-030a-43ca-a04c-bf00a788c3d6 nodeName:}" failed. No retries permitted until 2026-11-01 05:42:23.76502784 +0000 UTC m=+2358.905616760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6") : [secret "agc-cred-2e04ffa38ad1" not found, secret "agc-cred-388566305a92" not found, secret "agc-cred-500efa28b76e" not found] Nov 01 05:42:08.710399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:08.710368 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:42:08.711427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:08.711398 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:42:10.602391 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:10.602356 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:10.602391 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:10.602386 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:10.602391 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:10.602395 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:10.602874 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:10.602407 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:10.602874 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:10.602480 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:18.602458571 +0000 UTC m=+2353.743047473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:14.097432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:14.097390 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:42:15.588459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:15.588412 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:42:18.467136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:18.466960 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:42:18.494002 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:18.493961 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:42:18.495443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:18.495411 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:42:18.633169 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633132 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-17c481ce95fa: secret "agc-cred-17c481ce95fa" not found Nov 01 05:42:18.633169 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633161 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-31f7d618cb24: secret "agc-cred-31f7d618cb24" not found Nov 01 05:42:18.633169 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633169 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found Nov 01 05:42:18.633445 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633184 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-8eb61444db22: secret "agc-cred-8eb61444db22" not found Nov 01 05:42:18.633445 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633196 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs: [secret "agc-cred-17c481ce95fa" not found, secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:18.633445 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:18.633278 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials podName:338459c8-67a4-4e10-8a9b-8c5fa963d39e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:34.633235248 +0000 UTC m=+2369.773824144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e") : [secret "agc-cred-17c481ce95fa" not found, secret "agc-cred-31f7d618cb24" not found, secret "agc-cred-3e39090bc554" not found, secret "agc-cred-8eb61444db22" not found] Nov 01 05:42:20.589358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.589318 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" podUID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" containerName="payload-processing" containerID="cri-o://157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce" gracePeriod=25 Nov 01 05:42:20.877338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.877312 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:42:20.985883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.985847 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\") pod \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " Nov 01 05:42:20.986154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986137 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\") pod \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " Nov 01 05:42:20.986286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986194 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\") pod \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " Nov 01 05:42:20.986286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986214 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume" pod "338459c8-67a4-4e10-8a9b-8c5fa963d39e" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:20.986286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986238 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\") pod \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\" (UID: \"338459c8-67a4-4e10-8a9b-8c5fa963d39e\") " Nov 01 05:42:20.986825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986656 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume" pod "338459c8-67a4-4e10-8a9b-8c5fa963d39e" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:20.987101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986866 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:20.987101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.986886 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338459c8-67a4-4e10-8a9b-8c5fa963d39e-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:20.988669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.988645 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials" pod "338459c8-67a4-4e10-8a9b-8c5fa963d39e" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:20.988757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:20.988684 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww" pod "338459c8-67a4-4e10-8a9b-8c5fa963d39e" (UID: "338459c8-67a4-4e10-8a9b-8c5fa963d39e"). InnerVolumeSpecName "kube-api-access-jd5ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:21.089200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.089158 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5ww\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-kube-api-access-jd5ww\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:21.089200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.089198 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/338459c8-67a4-4e10-8a9b-8c5fa963d39e-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:21.361149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.361105 4425 generic.go:356] "Generic (PLEG): container finished" podID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" containerID="157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce" exitCode=0 Nov 01 05:42:21.361358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.361189 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" Nov 01 05:42:21.361358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.361199 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" event={"ID":"338459c8-67a4-4e10-8a9b-8c5fa963d39e","Type":"ContainerDied","Data":"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce"} Nov 01 05:42:21.361358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.361328 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs" event={"ID":"338459c8-67a4-4e10-8a9b-8c5fa963d39e","Type":"ContainerDied","Data":"490d64d460ea2ed1b14ce6ec9e220fd43534b1bbeec5f6f84c54722045a6cc13"} Nov 01 05:42:21.361491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.361366 4425 scope.go:122] "RemoveContainer" containerID="157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce" Nov 01 05:42:21.373224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.373200 4425 scope.go:122] "RemoveContainer" containerID="157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce" Nov 01 05:42:21.373635 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:21.373601 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce\": container with ID starting with 157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce not found: ID does not exist" containerID="157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce" Nov 01 05:42:21.373717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.373650 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce"} err="failed to get container status \"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce\": rpc error: code = NotFound desc = could not find container \"157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce\": container with ID starting with 157906b9b30ca18510c2936ba11dcbb8853eff926fa5562365a0eb9762e639ce not found: ID does not exist" Nov 01 05:42:21.529097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.529044 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:42:21.532913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:21.532874 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-676c64mjs"] Nov 01 05:42:22.977248 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:22.977215 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" path="/var/lib/kubelet/pods/338459c8-67a4-4e10-8a9b-8c5fa963d39e/volumes" Nov 01 05:42:23.468385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.468337 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" podUID="7f496e24-030a-43ca-a04c-bf00a788c3d6" containerName="payload-processing" containerID="cri-o://d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3" gracePeriod=25 Nov 01 05:42:23.494966 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.494909 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" containerID="cri-o://e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792" gracePeriod=25 Nov 01 05:42:23.792024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.791990 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:42:23.824704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.824675 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\") pod \"7f496e24-030a-43ca-a04c-bf00a788c3d6\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " Nov 01 05:42:23.824867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.824730 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\") pod \"7f496e24-030a-43ca-a04c-bf00a788c3d6\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " Nov 01 05:42:23.824867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.824784 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\") pod \"7f496e24-030a-43ca-a04c-bf00a788c3d6\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " Nov 01 05:42:23.825018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.824999 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\") pod \"7f496e24-030a-43ca-a04c-bf00a788c3d6\" (UID: \"7f496e24-030a-43ca-a04c-bf00a788c3d6\") " Nov 01 05:42:23.825536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.825309 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume" pod "7f496e24-030a-43ca-a04c-bf00a788c3d6" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:23.825536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.825511 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:23.825702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.825621 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume" pod "7f496e24-030a-43ca-a04c-bf00a788c3d6" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:23.827060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.827037 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws" pod "7f496e24-030a-43ca-a04c-bf00a788c3d6" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6"). InnerVolumeSpecName "kube-api-access-whpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:23.827146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.827108 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials" pod "7f496e24-030a-43ca-a04c-bf00a788c3d6" (UID: "7f496e24-030a-43ca-a04c-bf00a788c3d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:23.827394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.827378 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:42:23.926269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926218 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\") pod \"0d53569d-5233-4b9c-a20f-509e3ff68b13\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926303 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\") pod \"0d53569d-5233-4b9c-a20f-509e3ff68b13\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926487 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\") pod \"0d53569d-5233-4b9c-a20f-509e3ff68b13\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926526 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\") pod \"0d53569d-5233-4b9c-a20f-509e3ff68b13\" (UID: \"0d53569d-5233-4b9c-a20f-509e3ff68b13\") " Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926917 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume" pod "0d53569d-5233-4b9c-a20f-509e3ff68b13" (UID: "0d53569d-5233-4b9c-a20f-509e3ff68b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926974 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.926994 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpws\" (UniqueName: \"kubernetes.io/projected/7f496e24-030a-43ca-a04c-bf00a788c3d6-kube-api-access-whpws\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.927009 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f496e24-030a-43ca-a04c-bf00a788c3d6-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:23.928659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.927213 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume" pod "0d53569d-5233-4b9c-a20f-509e3ff68b13" (UID: "0d53569d-5233-4b9c-a20f-509e3ff68b13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:23.929191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.929164 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np" pod "0d53569d-5233-4b9c-a20f-509e3ff68b13" (UID: "0d53569d-5233-4b9c-a20f-509e3ff68b13"). InnerVolumeSpecName "kube-api-access-fh4np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:23.929322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:23.929271 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials" pod "0d53569d-5233-4b9c-a20f-509e3ff68b13" (UID: "0d53569d-5233-4b9c-a20f-509e3ff68b13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:24.028096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.028060 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:24.028096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.028089 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d53569d-5233-4b9c-a20f-509e3ff68b13-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:24.028096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.028098 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4np\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-kube-api-access-fh4np\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:24.028629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.028107 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d53569d-5233-4b9c-a20f-509e3ff68b13-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:24.386484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.386450 4425 generic.go:356] "Generic (PLEG): container finished" podID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerID="e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792" exitCode=0 Nov 01 05:42:24.386698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.386507 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" Nov 01 05:42:24.386698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.386527 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" event={"ID":"0d53569d-5233-4b9c-a20f-509e3ff68b13","Type":"ContainerDied","Data":"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792"} Nov 01 05:42:24.386698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.386580 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf" event={"ID":"0d53569d-5233-4b9c-a20f-509e3ff68b13","Type":"ContainerDied","Data":"a4ee251d83933822afa037060cc392f3d9199053f6a9d65e65a47a706713d885"} Nov 01 05:42:24.386698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.386601 4425 scope.go:122] "RemoveContainer" containerID="e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792" Nov 01 05:42:24.388671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.388644 4425 generic.go:356] "Generic (PLEG): container finished" podID="a6c211a5-586a-4f41-91b7-738cae284326" containerID="8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3" exitCode=0 Nov 01 05:42:24.388811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.388715 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerDied","Data":"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3"} Nov 01 05:42:24.389201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.389182 4425 scope.go:122] "RemoveContainer" containerID="8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3" Nov 01 05:42:24.390535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.390511 4425 generic.go:356] "Generic (PLEG): container finished" podID="7f496e24-030a-43ca-a04c-bf00a788c3d6" containerID="d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3" exitCode=0 Nov 01 05:42:24.390703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.390546 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" event={"ID":"7f496e24-030a-43ca-a04c-bf00a788c3d6","Type":"ContainerDied","Data":"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3"} Nov 01 05:42:24.390703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.390561 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" Nov 01 05:42:24.390703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.390588 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf" event={"ID":"7f496e24-030a-43ca-a04c-bf00a788c3d6","Type":"ContainerDied","Data":"865784bea716737206e0128b81d3ddab0f961f69be39a42a9d01adf38fe7cd13"} Nov 01 05:42:24.391192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.391157 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:24.391298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.391272 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:24.391356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.391328 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:42:24.399932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.399910 4425 scope.go:122] "RemoveContainer" containerID="e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792" Nov 01 05:42:24.400506 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:24.400480 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792\": container with ID starting with e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792 not found: ID does not exist" containerID="e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792" Nov 01 05:42:24.400609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.400518 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792"} err="failed to get container status \"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792\": rpc error: code = NotFound desc = could not find container \"e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792\": container with ID starting with e435e28f3392d0aa0c7ae971c26678ffb0d3e838bcbc2dd71310e47e18ba8792 not found: ID does not exist" Nov 01 05:42:24.400609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.400543 4425 scope.go:122] "RemoveContainer" containerID="d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3" Nov 01 05:42:24.412951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.412926 4425 scope.go:122] "RemoveContainer" containerID="d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3" Nov 01 05:42:24.413356 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:24.413269 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3\": container with ID starting with d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3 not found: ID does not exist" containerID="d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3" Nov 01 05:42:24.413356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.413306 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3"} err="failed to get container status \"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3\": rpc error: code = NotFound desc = could not find container \"d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3\": container with ID starting with d2147c8fe9f46820cf6a87bb945569d0ca8e7c3926bb2088b6154894021191a3 not found: ID does not exist" Nov 01 05:42:24.556080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.556041 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:42:24.562106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.562075 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db84j8xf"] Nov 01 05:42:24.802117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.802012 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:42:24.805462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.805423 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c7czk2jf"] Nov 01 05:42:24.985197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.984237 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" path="/var/lib/kubelet/pods/0d53569d-5233-4b9c-a20f-509e3ff68b13/volumes" Nov 01 05:42:24.985852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:24.985822 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f496e24-030a-43ca-a04c-bf00a788c3d6" path="/var/lib/kubelet/pods/7f496e24-030a-43ca-a04c-bf00a788c3d6/volumes" Nov 01 05:42:25.400033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:25.400003 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerStarted","Data":"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7"} Nov 01 05:42:25.925548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:25.925514 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:42:25.925756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:25.925694 4425 patch_prober.go:29] interesting pod/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 05:42:25.925756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:25.925741 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 05:42:26.925245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:26.925216 4425 patch_prober.go:29] interesting pod/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 05:42:26.925652 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:26.925296 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 05:42:26.972599 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:26.972552 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:42:27.932629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:27.932592 4425 patch_prober.go:29] interesting pod/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 05:42:27.933236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:27.933200 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 05:42:28.458156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.458116 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh"] Nov 01 05:42:28.459515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.459485 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" Nov 01 05:42:28.459694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.459679 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" Nov 01 05:42:28.459819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.459807 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7f496e24-030a-43ca-a04c-bf00a788c3d6" containerName="payload-processing" Nov 01 05:42:28.459900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.459889 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f496e24-030a-43ca-a04c-bf00a788c3d6" containerName="payload-processing" Nov 01 05:42:28.459988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.459978 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d945ff35-d6e1-400c-bc12-89f7175e9abc" containerName="revoke-keys" Nov 01 05:42:28.460062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.460052 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d945ff35-d6e1-400c-bc12-89f7175e9abc" containerName="revoke-keys" Nov 01 05:42:28.460151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.460141 4425 cpu_manager.go:414] "containerMap: removing container" podUID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" containerName="payload-processing" Nov 01 05:42:28.460224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.460215 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" containerName="payload-processing" Nov 01 05:42:28.460503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.460485 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d53569d-5233-4b9c-a20f-509e3ff68b13" containerName="payload-processing" Nov 01 05:42:28.460618 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.460606 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f496e24-030a-43ca-a04c-bf00a788c3d6" containerName="payload-processing" Nov 01 05:42:28.464315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.464293 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d945ff35-d6e1-400c-bc12-89f7175e9abc" containerName="revoke-keys" Nov 01 05:42:28.464461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.464450 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="338459c8-67a4-4e10-8a9b-8c5fa963d39e" containerName="payload-processing" Nov 01 05:42:28.525365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.523474 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh"] Nov 01 05:42:28.525365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.523592 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.577341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.577309 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-dc7d983de1b3\"" Nov 01 05:42:28.577517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.577376 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-rd4xw\"" Nov 01 05:42:28.620369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.620329 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpwn\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.620566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.620386 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.620640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.620569 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.620728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.620712 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.628128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.628102 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\"" Nov 01 05:42:28.628242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.628122 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\"" Nov 01 05:42:28.628242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.628107 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\"" Nov 01 05:42:28.628242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.628177 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\"" Nov 01 05:42:28.721998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.721904 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpwn\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.721998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.721959 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.722285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.722079 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.722285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.722168 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.722414 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:28.722391 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:28.722414 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:28.722410 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:28.722524 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:28.722489 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:29.222468481 +0000 UTC m=+2364.363057393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:28.722862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.722833 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.723474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.723452 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.786346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.786311 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpwn\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:28.926787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.926753 4425 patch_prober.go:29] interesting pod/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 05:42:28.927105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:28.927061 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 05:42:29.228827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:29.228790 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:29.229283 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:29.228939 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:29.229283 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:29.228956 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:29.229283 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:29.229009 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:30.228994605 +0000 UTC m=+2365.369583523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:29.926030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:29.925999 4425 patch_prober.go:29] interesting pod/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 05:42:29.926213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:29.926062 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 05:42:29.978766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:29.978727 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:42:29.983134 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:29.983115 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.039341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.039301 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"e2e-cfl-eafa67-provider-cred\"" Nov 01 05:42:30.040033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.040001 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.040369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.040330 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.040482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.040383 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.040580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.040562 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsjf\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.056280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.042148 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:42:30.158291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.141506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.158653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.158618 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.158939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.158916 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsjf\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.163342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.163310 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.166912 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.166886 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.167868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.167522 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.169757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.169738 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.227630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.227553 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsjf\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.267350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.267305 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:30.267798 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:30.267505 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:30.267798 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:30.267524 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:30.267798 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:30.267587 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:32.267567995 +0000 UTC m=+2367.408156894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:30.297382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.297345 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:30.706097 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:42:30.706062 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7801c258_cc25_42ba_baf4_6cdf0b07d67c.slice/crio-d02327afc71729b56c7178b9852b010bb9a91768f411973671618d7211b65388 WatchSource:0}: Error finding container d02327afc71729b56c7178b9852b010bb9a91768f411973671618d7211b65388: Status 404 returned error can't find the container with id d02327afc71729b56c7178b9852b010bb9a91768f411973671618d7211b65388 Nov 01 05:42:30.707779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.707746 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:42:30.930797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.930757 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:42:30.931085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.931068 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:42:30.931787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:30.931773 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:42:31.448442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:31.448398 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" event={"ID":"7801c258-cc25-42ba-baf4-6cdf0b07d67c","Type":"ContainerStarted","Data":"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5"} Nov 01 05:42:31.448856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:31.448464 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" event={"ID":"7801c258-cc25-42ba-baf4-6cdf0b07d67c","Type":"ContainerStarted","Data":"d02327afc71729b56c7178b9852b010bb9a91768f411973671618d7211b65388"} Nov 01 05:42:31.448856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:31.448490 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:42:31.502189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:31.502158 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:42:31.567246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:31.567189 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podStartSLOduration=2.567173854 podStartE2EDuration="2.567173854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:42:31.566593153 +0000 UTC m=+2366.707182112" watchObservedRunningTime="2026-11-01 05:42:31.567173854 +0000 UTC m=+2366.707762751" Nov 01 05:42:32.294271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:32.294219 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:32.294515 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:32.294398 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:32.294515 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:32.294423 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:32.294515 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:32.294496 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:36.294475751 +0000 UTC m=+2371.435064662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:36.365968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.365903 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:36.366453 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:36.366045 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:36.366453 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:36.366065 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:36.366453 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:36.366119 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:42:44.366104973 +0000 UTC m=+2379.506693882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:36.502629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.502589 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" podUID="f6d76782-0b88-4948-b199-5a12d13da32b" containerName="payload-pre-processing" containerID="cri-o://1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600" gracePeriod=25 Nov 01 05:42:36.784549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.784517 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:42:36.872650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.872556 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\") pod \"f6d76782-0b88-4948-b199-5a12d13da32b\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " Nov 01 05:42:36.872851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.872733 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\" (UniqueName: \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\") pod \"f6d76782-0b88-4948-b199-5a12d13da32b\" (UID: \"f6d76782-0b88-4948-b199-5a12d13da32b\") " Nov 01 05:42:36.873031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.873005 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume" pod "f6d76782-0b88-4948-b199-5a12d13da32b" (UID: "f6d76782-0b88-4948-b199-5a12d13da32b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:42:36.873352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.873331 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d76782-0b88-4948-b199-5a12d13da32b-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:36.875083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.875049 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6" pod "f6d76782-0b88-4948-b199-5a12d13da32b" (UID: "f6d76782-0b88-4948-b199-5a12d13da32b"). InnerVolumeSpecName "kube-api-access-4h2q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:42:36.974035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:36.973999 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2q6\" (UniqueName: \"kubernetes.io/projected/f6d76782-0b88-4948-b199-5a12d13da32b-kube-api-access-4h2q6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:42:37.490445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.490404 4425 generic.go:356] "Generic (PLEG): container finished" podID="f6d76782-0b88-4948-b199-5a12d13da32b" containerID="1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600" exitCode=0 Nov 01 05:42:37.490926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.490466 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" Nov 01 05:42:37.490926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.490466 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" event={"ID":"f6d76782-0b88-4948-b199-5a12d13da32b","Type":"ContainerDied","Data":"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600"} Nov 01 05:42:37.490926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.490582 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb" event={"ID":"f6d76782-0b88-4948-b199-5a12d13da32b","Type":"ContainerDied","Data":"3606fb0336e66b148defe3fbac45f7c9dc1431ce93fba6f52a9d1675b004cdfa"} Nov 01 05:42:37.490926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.490615 4425 scope.go:122] "RemoveContainer" containerID="1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600" Nov 01 05:42:37.503626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.503368 4425 scope.go:122] "RemoveContainer" containerID="1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600" Nov 01 05:42:37.503865 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:37.503842 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600\": container with ID starting with 1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600 not found: ID does not exist" containerID="1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600" Nov 01 05:42:37.503944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.503874 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600"} err="failed to get container status \"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600\": rpc error: code = NotFound desc = could not find container \"1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600\": container with ID starting with 1e872049a03c34b938798a80d359f1fdcfd09a4c6442e4b35878943671c62600 not found: ID does not exist" Nov 01 05:42:37.660358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.660322 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:42:37.667538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:37.667506 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a08eb536-6fc8fc647-2pmhb"] Nov 01 05:42:38.981774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:38.981733 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d76782-0b88-4948-b199-5a12d13da32b" path="/var/lib/kubelet/pods/f6d76782-0b88-4948-b199-5a12d13da32b/volumes" Nov 01 05:42:42.459554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:42.459515 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:42:44.374740 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:44.374698 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:42:44.375172 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:44.374870 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:44.375172 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:44.374890 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:44.375172 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:44.374957 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:43:00.374941986 +0000 UTC m=+2395.515530894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:42:52.458531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:52.458442 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:42:56.949401 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:42:56.949360 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:42:58.668934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.668902 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:42:58.669619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.669601 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f6d76782-0b88-4948-b199-5a12d13da32b" containerName="payload-pre-processing" Nov 01 05:42:58.669667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.669621 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d76782-0b88-4948-b199-5a12d13da32b" containerName="payload-pre-processing" Nov 01 05:42:58.669757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.669747 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d76782-0b88-4948-b199-5a12d13da32b" containerName="payload-pre-processing" Nov 01 05:42:58.674448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.674429 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.731243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.731211 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:42:58.793311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.786594 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8815d2\"" Nov 01 05:42:58.793311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.786764 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8815d2-dockercfg-rwv6p\"" Nov 01 05:42:58.793311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.789403 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.793311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.789558 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7nc\" (UniqueName: \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.891143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.891110 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7nc\" (UniqueName: \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.891353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.891245 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.891961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.891936 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.953865 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.953787 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7nc\" (UniqueName: \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\") pod \"payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:58.988903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:58.988866 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:59.368367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:59.368333 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:42:59.369154 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:42:59.369125 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb768b926_d65b_4ed7_afc0_bf5111096264.slice/crio-950381cffc69562eb8a7dec53d5a5fdd42eeb9103a15caa7aed8faca7f488f64 WatchSource:0}: Error finding container 950381cffc69562eb8a7dec53d5a5fdd42eeb9103a15caa7aed8faca7f488f64: Status 404 returned error can't find the container with id 950381cffc69562eb8a7dec53d5a5fdd42eeb9103a15caa7aed8faca7f488f64 Nov 01 05:42:59.657110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:59.657011 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" event={"ID":"b768b926-d65b-4ed7-afc0-bf5111096264","Type":"ContainerStarted","Data":"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9"} Nov 01 05:42:59.657110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:59.657089 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" event={"ID":"b768b926-d65b-4ed7-afc0-bf5111096264","Type":"ContainerStarted","Data":"950381cffc69562eb8a7dec53d5a5fdd42eeb9103a15caa7aed8faca7f488f64"} Nov 01 05:42:59.657330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:59.657123 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:42:59.785581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:42:59.785533 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" podStartSLOduration=1.7855181359999999 podStartE2EDuration="1.785518136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:42:59.783472406 +0000 UTC m=+2394.924061325" watchObservedRunningTime="2026-11-01 05:42:59.785518136 +0000 UTC m=+2394.926107093" Nov 01 05:43:00.412557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:00.412521 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:43:00.412773 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:00.412678 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:00.412773 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:00.412692 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:00.412773 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:00.412751 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:43:32.412733031 +0000 UTC m=+2427.553321946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:02.458335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:02.458295 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:06.829579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.829544 4425 scope.go:122] "RemoveContainer" containerID="e11619ea82b5f2d4f6201a7e0b828ad1e03be59cb1b17797605ad261c8c7cfd6" Nov 01 05:43:06.921997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.921961 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:43:06.922282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.922231 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" containerID="cri-o://977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b" gracePeriod=30 Nov 01 05:43:06.975411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.975374 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:43:06.975634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.975611 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" containerID="cri-o://40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f" gracePeriod=30 Nov 01 05:43:06.977382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.977360 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:43:06.977557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:06.977524 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" containerID="cri-o://d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816" gracePeriod=30 Nov 01 05:43:07.023413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:07.023379 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:43:07.023614 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:07.023591 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" containerID="cri-o://0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513" gracePeriod=30 Nov 01 05:43:10.438535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.438497 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k"] Nov 01 05:43:10.444504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.444482 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.500603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.500575 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:43:10.503081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.503053 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k"] Nov 01 05:43:10.544651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.544616 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.544834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.544684 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvkz\" (UniqueName: \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.544951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.544924 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.548564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.548542 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:43:10.548685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.548573 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:43:10.551300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.551276 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:43:10.645972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.645937 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.646158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.646029 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvkz\" (UniqueName: \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.646284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.646246 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.646593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.646571 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.648741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.648720 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.667841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.667816 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:43:10.708601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.708516 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvkz\" (UniqueName: \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\") pod \"maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:10.758544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:10.758515 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:11.149749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:11.149716 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k"] Nov 01 05:43:11.150303 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:11.150273 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf815a7_c470_428d_8b20_d7fdfcf98761.slice/crio-cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880 WatchSource:0}: Error finding container cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880: Status 404 returned error can't find the container with id cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880 Nov 01 05:43:11.744726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:11.744626 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" event={"ID":"bcf815a7-c470-428d-8b20-d7fdfcf98761","Type":"ContainerStarted","Data":"5b91afd8ae702b3ae558d5eea896e4794650bf0cad300ca9e3f90455115eb471"} Nov 01 05:43:11.744726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:11.744696 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" event={"ID":"bcf815a7-c470-428d-8b20-d7fdfcf98761","Type":"ContainerStarted","Data":"cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880"} Nov 01 05:43:11.862222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:11.862172 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" podStartSLOduration=1.862157064 podStartE2EDuration="1.862157064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:11.86132473 +0000 UTC m=+2407.001913650" watchObservedRunningTime="2026-11-01 05:43:11.862157064 +0000 UTC m=+2407.002745982" Nov 01 05:43:12.248276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.248231 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:43:12.341216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.341183 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:43:12.366818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.366793 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:43:12.369667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369646 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.369814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369688 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.369814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369712 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.369814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369750 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.369814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369770 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.370657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369923 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.370657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.369983 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.370657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.370031 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.370657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.370195 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371071 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371218 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371313 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371402 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371436 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371524 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371621 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371654 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371683 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\") pod \"49c9f4f7-d918-4d97-8e43-ac63af55081e\" (UID: \"49c9f4f7-d918-4d97-8e43-ac63af55081e\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371715 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371752 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.371811 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\") pod \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\" (UID: \"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533\") " Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.372364 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.372462 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.373059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.372566 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.374365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.374100 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.374365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.374335 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.375163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375132 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:12.375304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375207 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375231 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375245 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375313 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c9f4f7-d918-4d97-8e43-ac63af55081e-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375326 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375339 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.375585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.375353 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.376242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.376216 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "kube-api-access-2hngg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.376397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.376334 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.376486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.376391 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.376486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.376351 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.376486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.376452 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.377229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.377191 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:43:12.377229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.377198 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "kube-api-access-z6wzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.377748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.377714 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.377947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.377926 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy" pod "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" (UID: "7bd74491-1ceb-4ef4-b5ef-a0a226cd2533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.378384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.378364 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo" pod "49c9f4f7-d918-4d97-8e43-ac63af55081e" (UID: "49c9f4f7-d918-4d97-8e43-ac63af55081e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:43:12.458107 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.458069 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:12.476430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476400 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476440 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476462 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476485 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476541 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476579 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476603 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476644 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476670 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476702 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476724 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476750 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476797 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476818 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476848 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.476870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476865 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.477629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476924 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.477629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476963 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\") pod \"5eb24c12-7422-4b1d-9977-214836baf59a\" (UID: \"5eb24c12-7422-4b1d-9977-214836baf59a\") " Nov 01 05:43:12.477629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.476997 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.477629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477029 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\") pod \"9254bc61-bbde-4309-b9ae-57c7859bee94\" (UID: \"9254bc61-bbde-4309-b9ae-57c7859bee94\") " Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477704 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wzw\" (UniqueName: \"kubernetes.io/projected/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-kube-api-access-z6wzw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477728 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477743 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477757 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hngg\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-kube-api-access-2hngg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477769 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477781 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477795 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477808 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477822 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.477846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477836 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c9f4f7-d918-4d97-8e43-ac63af55081e-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477852 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477868 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c9f4f7-d918-4d97-8e43-ac63af55081e-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477882 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477025 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477088 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477186 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477450 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477554 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.477872 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.478038 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.479218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.478565 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.482393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482358 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:43:12.482522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482491 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.482587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482553 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "kube-api-access-vlvpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.482645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482585 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "kube-api-access-nglpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.482645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482624 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.482733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482660 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:12.482874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.482856 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy" pod "9254bc61-bbde-4309-b9ae-57c7859bee94" (UID: "9254bc61-bbde-4309-b9ae-57c7859bee94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:43:12.483328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.483308 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo" pod "5eb24c12-7422-4b1d-9977-214836baf59a" (UID: "5eb24c12-7422-4b1d-9977-214836baf59a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:43:12.579461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579416 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579457 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579475 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579492 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579507 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579523 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9254bc61-bbde-4309-b9ae-57c7859bee94-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579537 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579551 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb24c12-7422-4b1d-9977-214836baf59a-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579566 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579581 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579596 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579611 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb24c12-7422-4b1d-9977-214836baf59a-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579628 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvpw\" (UniqueName: \"kubernetes.io/projected/5eb24c12-7422-4b1d-9977-214836baf59a-kube-api-access-vlvpw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579643 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb24c12-7422-4b1d-9977-214836baf59a-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579657 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9254bc61-bbde-4309-b9ae-57c7859bee94-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.579688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.579672 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglpq\" (UniqueName: \"kubernetes.io/projected/9254bc61-bbde-4309-b9ae-57c7859bee94-kube-api-access-nglpq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:12.755567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.755526 4425 generic.go:356] "Generic (PLEG): container finished" podID="bcf815a7-c470-428d-8b20-d7fdfcf98761" containerID="5b91afd8ae702b3ae558d5eea896e4794650bf0cad300ca9e3f90455115eb471" exitCode=0 Nov 01 05:43:12.756023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.755668 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" event={"ID":"bcf815a7-c470-428d-8b20-d7fdfcf98761","Type":"ContainerDied","Data":"5b91afd8ae702b3ae558d5eea896e4794650bf0cad300ca9e3f90455115eb471"} Nov 01 05:43:12.757517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.757483 4425 generic.go:356] "Generic (PLEG): container finished" podID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerID="0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513" exitCode=0 Nov 01 05:43:12.757664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.757527 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" event={"ID":"9254bc61-bbde-4309-b9ae-57c7859bee94","Type":"ContainerDied","Data":"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513"} Nov 01 05:43:12.757664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.757579 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" event={"ID":"9254bc61-bbde-4309-b9ae-57c7859bee94","Type":"ContainerDied","Data":"6f06941393938dce2c027ec607b068737001cb0771f87215395ed15ddb03da96"} Nov 01 05:43:12.757664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.757607 4425 scope.go:122] "RemoveContainer" containerID="0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513" Nov 01 05:43:12.757933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.757909 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t" Nov 01 05:43:12.759402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.759383 4425 generic.go:356] "Generic (PLEG): container finished" podID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerID="40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f" exitCode=0 Nov 01 05:43:12.759574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.759434 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" Nov 01 05:43:12.760053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.759450 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" event={"ID":"49c9f4f7-d918-4d97-8e43-ac63af55081e","Type":"ContainerDied","Data":"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f"} Nov 01 05:43:12.760053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.759749 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv" event={"ID":"49c9f4f7-d918-4d97-8e43-ac63af55081e","Type":"ContainerDied","Data":"9e79aa7943c707a440143c735f428107ec8a5bc3daef80018ef836d24ae380ea"} Nov 01 05:43:12.782410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.775053 4425 generic.go:356] "Generic (PLEG): container finished" podID="5eb24c12-7422-4b1d-9977-214836baf59a" containerID="d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816" exitCode=0 Nov 01 05:43:12.782410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.775245 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" event={"ID":"5eb24c12-7422-4b1d-9977-214836baf59a","Type":"ContainerDied","Data":"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816"} Nov 01 05:43:12.782410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.775362 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" event={"ID":"5eb24c12-7422-4b1d-9977-214836baf59a","Type":"ContainerDied","Data":"efa4a7f8364892dc1db17c7007bd463ca86ccce5b85d90113d081383e8da0b9d"} Nov 01 05:43:12.782410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.775464 4425 scope.go:122] "RemoveContainer" containerID="0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513" Nov 01 05:43:12.782410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.775627 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2" Nov 01 05:43:12.784550 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:12.784015 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513\": container with ID starting with 0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513 not found: ID does not exist" containerID="0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513" Nov 01 05:43:12.784550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.784063 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513"} err="failed to get container status \"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513\": rpc error: code = NotFound desc = could not find container \"0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513\": container with ID starting with 0f03b9ada9d6ffb9141fc20993a648e4bb0bd20f68f4c54c3a9e71d7b3d0b513 not found: ID does not exist" Nov 01 05:43:12.784550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.784087 4425 scope.go:122] "RemoveContainer" containerID="40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f" Nov 01 05:43:12.786939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.786902 4425 generic.go:356] "Generic (PLEG): container finished" podID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerID="977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b" exitCode=0 Nov 01 05:43:12.787047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.787027 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" event={"ID":"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533","Type":"ContainerDied","Data":"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b"} Nov 01 05:43:12.787151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.787090 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" event={"ID":"7bd74491-1ceb-4ef4-b5ef-a0a226cd2533","Type":"ContainerDied","Data":"3f1f3e6796443d34f79ea6a39d422a76529ba533083ed57a25790a64d3f6bcac"} Nov 01 05:43:12.787151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.787111 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc" Nov 01 05:43:12.796998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.796973 4425 scope.go:122] "RemoveContainer" containerID="40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f" Nov 01 05:43:12.797490 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:12.797459 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f\": container with ID starting with 40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f not found: ID does not exist" containerID="40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f" Nov 01 05:43:12.797569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.797503 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f"} err="failed to get container status \"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f\": rpc error: code = NotFound desc = could not find container \"40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f\": container with ID starting with 40e468d61878b4b8989888661049b67ceda1500d43091dd0ff4851b64103a59f not found: ID does not exist" Nov 01 05:43:12.797569 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.797528 4425 scope.go:122] "RemoveContainer" containerID="d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816" Nov 01 05:43:12.809051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.809008 4425 scope.go:122] "RemoveContainer" containerID="d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816" Nov 01 05:43:12.809438 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:12.809410 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816\": container with ID starting with d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816 not found: ID does not exist" containerID="d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816" Nov 01 05:43:12.809521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.809451 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816"} err="failed to get container status \"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816\": rpc error: code = NotFound desc = could not find container \"d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816\": container with ID starting with d99698da869235e4a7d9ad14c18c34a3653671152caca5f54d3120f45df5f816 not found: ID does not exist" Nov 01 05:43:12.809521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.809479 4425 scope.go:122] "RemoveContainer" containerID="977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b" Nov 01 05:43:12.821290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.821272 4425 scope.go:122] "RemoveContainer" containerID="977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b" Nov 01 05:43:12.821580 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:12.821558 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b\": container with ID starting with 977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b not found: ID does not exist" containerID="977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b" Nov 01 05:43:12.821650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.821592 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b"} err="failed to get container status \"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b\": rpc error: code = NotFound desc = could not find container \"977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b\": container with ID starting with 977293259fc96c6d0682b6dfa01afcbb77366166cc1187e28a494d61f27f604b not found: ID does not exist" Nov 01 05:43:12.989361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.989241 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:43:12.992604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:12.992580 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tmwv2"] Nov 01 05:43:13.103303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.103270 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:43:13.109225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.109194 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zzhvc"] Nov 01 05:43:13.219758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.219723 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:43:13.224550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.224521 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kdh6t"] Nov 01 05:43:13.336530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.336473 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:43:13.340039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.340014 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-gxmrv"] Nov 01 05:43:13.942342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.942317 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:13.998904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.998870 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\") pod \"bcf815a7-c470-428d-8b20-d7fdfcf98761\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " Nov 01 05:43:13.999079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.998984 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\" (UniqueName: \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\") pod \"bcf815a7-c470-428d-8b20-d7fdfcf98761\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " Nov 01 05:43:13.999079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.999052 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\") pod \"bcf815a7-c470-428d-8b20-d7fdfcf98761\" (UID: \"bcf815a7-c470-428d-8b20-d7fdfcf98761\") " Nov 01 05:43:13.999366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.999343 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca" pod "bcf815a7-c470-428d-8b20-d7fdfcf98761" (UID: "bcf815a7-c470-428d-8b20-d7fdfcf98761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:13.999500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:13.999483 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:14.001339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.001317 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca" pod "bcf815a7-c470-428d-8b20-d7fdfcf98761" (UID: "bcf815a7-c470-428d-8b20-d7fdfcf98761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:43:14.001441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.001428 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz" pod "bcf815a7-c470-428d-8b20-d7fdfcf98761" (UID: "bcf815a7-c470-428d-8b20-d7fdfcf98761"). InnerVolumeSpecName "kube-api-access-gxvkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:14.101276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.101229 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvkz\" (UniqueName: \"kubernetes.io/projected/bcf815a7-c470-428d-8b20-d7fdfcf98761-kube-api-access-gxvkz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:14.101444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.101282 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf815a7-c470-428d-8b20-d7fdfcf98761-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:14.805815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.805784 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" Nov 01 05:43:14.805815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.805803 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k" event={"ID":"bcf815a7-c470-428d-8b20-d7fdfcf98761","Type":"ContainerDied","Data":"cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880"} Nov 01 05:43:14.806036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.805849 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd4e06fcf5af829c47a61e70acfec75b247b58de9343fa39b5c9637f5f1a880" Nov 01 05:43:14.973532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.973487 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" path="/var/lib/kubelet/pods/49c9f4f7-d918-4d97-8e43-ac63af55081e/volumes" Nov 01 05:43:14.974155 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.974132 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" path="/var/lib/kubelet/pods/5eb24c12-7422-4b1d-9977-214836baf59a/volumes" Nov 01 05:43:14.974655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.974636 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" path="/var/lib/kubelet/pods/7bd74491-1ceb-4ef4-b5ef-a0a226cd2533/volumes" Nov 01 05:43:14.975464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:14.975444 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" path="/var/lib/kubelet/pods/9254bc61-bbde-4309-b9ae-57c7859bee94/volumes" Nov 01 05:43:22.203284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203233 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:43:22.203905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203869 4425 cpu_manager.go:414] "containerMap: removing container" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" Nov 01 05:43:22.203905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203881 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" Nov 01 05:43:22.203905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203906 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203912 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203923 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203929 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203942 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bcf815a7-c470-428d-8b20-d7fdfcf98761" containerName="revoke-keys" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203947 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf815a7-c470-428d-8b20-d7fdfcf98761" containerName="revoke-keys" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203960 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.203965 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.204051 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd74491-1ceb-4ef4-b5ef-a0a226cd2533" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.204061 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf815a7-c470-428d-8b20-d7fdfcf98761" containerName="revoke-keys" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.204069 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb24c12-7422-4b1d-9977-214836baf59a" containerName="istio-proxy" Nov 01 05:43:22.204076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.204076 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c9f4f7-d918-4d97-8e43-ac63af55081e" containerName="istio-proxy" Nov 01 05:43:22.204681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.204086 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9254bc61-bbde-4309-b9ae-57c7859bee94" containerName="istio-proxy" Nov 01 05:43:22.209173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.209145 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.268472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.268433 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:43:22.300937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.300898 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.300937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.300945 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.300971 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjkz\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301187 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301219 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301367 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301474 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301509 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.301595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.301583 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.313636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.313594 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25565930-openshift-default-dockercfg-f64wx\"" Nov 01 05:43:22.412588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412509 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412582 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412611 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412674 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412758 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412792 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.412826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412820 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjkz\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.413098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412965 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.413098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.412987 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.426097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.426058 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.426342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.426194 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.426342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.426229 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.426525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.426504 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.426608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.426557 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.429528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.429500 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.429662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.429508 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.458411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.458310 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:22.478242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.478210 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.478521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.478498 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjkz\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\") pod \"e2e-mt-25565930-openshift-default-64f5666649-rq68b\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.525326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.525284 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:22.962717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.962687 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:43:22.962946 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:22.962923 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c8ba49_5944_4176_97ba_2fc5d20f35a4.slice/crio-f73d24feff8fe2c15137864c1ad87e3d6e7fd9f7a4c09aad9cdc266a5bf06e57 WatchSource:0}: Error finding container f73d24feff8fe2c15137864c1ad87e3d6e7fd9f7a4c09aad9cdc266a5bf06e57: Status 404 returned error can't find the container with id f73d24feff8fe2c15137864c1ad87e3d6e7fd9f7a4c09aad9cdc266a5bf06e57 Nov 01 05:43:22.965705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.965672 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:43:22.965784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.965757 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:43:22.965822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:22.965788 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:43:23.870179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:23.870140 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerStarted","Data":"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249"} Nov 01 05:43:23.870689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:23.870197 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerStarted","Data":"f73d24feff8fe2c15137864c1ad87e3d6e7fd9f7a4c09aad9cdc266a5bf06e57"} Nov 01 05:43:24.045709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:24.045652 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podStartSLOduration=2.045638483 podStartE2EDuration="2.045638483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:24.042576465 +0000 UTC m=+2419.183165400" watchObservedRunningTime="2026-11-01 05:43:24.045638483 +0000 UTC m=+2419.186227402" Nov 01 05:43:24.526341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:24.526306 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:24.532661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:24.532636 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:24.880859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:24.880822 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:24.892443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:24.892412 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:43:26.949286 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:26.949212 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:43:30.491340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.491305 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:43:30.495730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.495708 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.552387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.552355 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:43:30.601291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.600238 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25565930\"" Nov 01 05:43:30.601291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.600416 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25565930-dockercfg-jl5zk\"" Nov 01 05:43:30.623232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.623178 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5l9\" (UniqueName: \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.623500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.623407 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.655242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.655199 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:43:30.660374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.660348 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.698510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.698474 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:43:30.725170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.725142 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfnq\" (UniqueName: \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.725366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.725280 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5l9\" (UniqueName: \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.725366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.725340 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.725468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.725401 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.726132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.726112 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.787610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.787539 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5l9\" (UniqueName: \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.810507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.810475 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:30.826366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.826325 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfnq\" (UniqueName: \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.826534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.826453 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.827141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.827117 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.888635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.888581 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfnq\" (UniqueName: \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:30.975197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:30.975166 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:31.190890 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:31.190861 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c457594_ca7c_4176_b445_3995642442aa.slice/crio-a808a4f8a2425ad1532389d72a213724b1fdd51c4c1fc6f964b51111e740d083 WatchSource:0}: Error finding container a808a4f8a2425ad1532389d72a213724b1fdd51c4c1fc6f964b51111e740d083: Status 404 returned error can't find the container with id a808a4f8a2425ad1532389d72a213724b1fdd51c4c1fc6f964b51111e740d083 Nov 01 05:43:31.191846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.191813 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:43:31.356783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.356753 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:43:31.358323 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:31.358291 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadeb7d4_ac45_4958_880a_56bc93e80682.slice/crio-bfeb6ed4a5b8fcfe525af53341fa1c4dcc0f04d2c172a2a8ab07ee3d77274e7b WatchSource:0}: Error finding container bfeb6ed4a5b8fcfe525af53341fa1c4dcc0f04d2c172a2a8ab07ee3d77274e7b: Status 404 returned error can't find the container with id bfeb6ed4a5b8fcfe525af53341fa1c4dcc0f04d2c172a2a8ab07ee3d77274e7b Nov 01 05:43:31.928137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.928097 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" event={"ID":"fadeb7d4-ac45-4958-880a-56bc93e80682","Type":"ContainerStarted","Data":"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91"} Nov 01 05:43:31.928581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.928166 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" event={"ID":"fadeb7d4-ac45-4958-880a-56bc93e80682","Type":"ContainerStarted","Data":"bfeb6ed4a5b8fcfe525af53341fa1c4dcc0f04d2c172a2a8ab07ee3d77274e7b"} Nov 01 05:43:31.928581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.928285 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:31.929974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.929949 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" event={"ID":"5c457594-ca7c-4176-b445-3995642442aa","Type":"ContainerStarted","Data":"77740e5f9f6056aee84fe339b115cfa1e8d8860f535751e279825c6a8b24ade0"} Nov 01 05:43:31.930109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.930007 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" event={"ID":"5c457594-ca7c-4176-b445-3995642442aa","Type":"ContainerStarted","Data":"a808a4f8a2425ad1532389d72a213724b1fdd51c4c1fc6f964b51111e740d083"} Nov 01 05:43:31.930109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:31.930036 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:32.053725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:32.053663 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" podStartSLOduration=2.053647679 podStartE2EDuration="2.053647679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:32.050370477 +0000 UTC m=+2427.190959415" watchObservedRunningTime="2026-11-01 05:43:32.053647679 +0000 UTC m=+2427.194236651" Nov 01 05:43:32.170504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:32.170443 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" podStartSLOduration=2.170422513 podStartE2EDuration="2.170422513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:32.168359105 +0000 UTC m=+2427.308948026" watchObservedRunningTime="2026-11-01 05:43:32.170422513 +0000 UTC m=+2427.311011431" Nov 01 05:43:32.445131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:32.445099 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:43:32.445311 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:32.445282 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:32.445311 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:32.445303 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:32.445417 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:32.445360 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:44:36.445343819 +0000 UTC m=+2491.585932742 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:43:32.458333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:32.458294 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:37.149182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.149138 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:43:37.153961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.153936 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.210540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.210503 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:43:37.210880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.210843 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.211090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.211073 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.211153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.211132 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.211221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.211204 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmqw\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.312891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.312844 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.313091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.312916 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.313091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.312961 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmqw\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.313091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.312995 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.313671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.313640 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.313808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.313785 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.315785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.315761 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.375135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.375101 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmqw\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.469361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.469270 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:37.849057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.849027 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:43:37.849862 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:37.849835 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89ddc5e_fa32_4226_9f97_e446c54aae84.slice/crio-5250d523e17a29647f8bf95fd1fc04cadb23c5fa04e7f1182a8c3ee1d049e12e WatchSource:0}: Error finding container 5250d523e17a29647f8bf95fd1fc04cadb23c5fa04e7f1182a8c3ee1d049e12e: Status 404 returned error can't find the container with id 5250d523e17a29647f8bf95fd1fc04cadb23c5fa04e7f1182a8c3ee1d049e12e Nov 01 05:43:37.971053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.971014 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" event={"ID":"c89ddc5e-fa32-4226-9f97-e446c54aae84","Type":"ContainerStarted","Data":"14e403c870a1e94b10d2bd6f5e4ff537460bc154955fd13f8929e590fad0cbe8"} Nov 01 05:43:37.971245 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.971105 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" event={"ID":"c89ddc5e-fa32-4226-9f97-e446c54aae84","Type":"ContainerStarted","Data":"5250d523e17a29647f8bf95fd1fc04cadb23c5fa04e7f1182a8c3ee1d049e12e"} Nov 01 05:43:37.971348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:37.971270 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:43:38.090954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:38.090904 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podStartSLOduration=1.090890751 podStartE2EDuration="1.090890751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:38.090073881 +0000 UTC m=+2433.230662802" watchObservedRunningTime="2026-11-01 05:43:38.090890751 +0000 UTC m=+2433.231479670" Nov 01 05:43:38.967431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:38.967392 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:40.932960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:40.932928 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr"] Nov 01 05:43:40.937398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:40.937381 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:40.995814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:40.995787 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr"] Nov 01 05:43:41.041745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.041717 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:43:41.041745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.041717 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:43:41.041745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.041727 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:43:41.041994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.041766 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:43:41.059353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.059319 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8pn\" (UniqueName: \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.059535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.059398 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.059535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.059481 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.160816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.160778 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8pn\" (UniqueName: \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.160970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.160858 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.160970 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.160888 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.161572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.161550 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.163572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.163547 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.223129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.223048 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8pn\" (UniqueName: \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\") pod \"maas-api-revoke-sub-f9578fbd-k6nmr\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.251947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.251913 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:41.629444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:41.629411 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr"] Nov 01 05:43:41.630044 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:41.630004 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721b275d_d9b9_44a1_83b7_3d2b33f22dbe.slice/crio-ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403 WatchSource:0}: Error finding container ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403: Status 404 returned error can't find the container with id ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403 Nov 01 05:43:42.000891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:42.000854 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" event={"ID":"721b275d-d9b9-44a1-83b7-3d2b33f22dbe","Type":"ContainerStarted","Data":"6e604ffb0beeeabcd11a1b2cfed2b122e27c5658d98440b7b24e0fae25baf1f1"} Nov 01 05:43:42.001400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:42.000915 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" event={"ID":"721b275d-d9b9-44a1-83b7-3d2b33f22dbe","Type":"ContainerStarted","Data":"ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403"} Nov 01 05:43:42.120131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:42.120072 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" podStartSLOduration=2.120057322 podStartE2EDuration="2.120057322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:42.117694783 +0000 UTC m=+2437.258283726" watchObservedRunningTime="2026-11-01 05:43:42.120057322 +0000 UTC m=+2437.260646241" Nov 01 05:43:42.940280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:42.940174 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:43:42.940280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:42.940247 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:43:43.009034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:43.009004 4425 generic.go:356] "Generic (PLEG): container finished" podID="721b275d-d9b9-44a1-83b7-3d2b33f22dbe" containerID="6e604ffb0beeeabcd11a1b2cfed2b122e27c5658d98440b7b24e0fae25baf1f1" exitCode=0 Nov 01 05:43:43.009453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:43.009108 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" event={"ID":"721b275d-d9b9-44a1-83b7-3d2b33f22dbe","Type":"ContainerDied","Data":"6e604ffb0beeeabcd11a1b2cfed2b122e27c5658d98440b7b24e0fae25baf1f1"} Nov 01 05:43:44.196784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.196756 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:44.302716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.302634 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\") pod \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " Nov 01 05:43:44.302878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.302728 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\" (UniqueName: \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\") pod \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " Nov 01 05:43:44.302878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.302807 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\") pod \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\" (UID: \"721b275d-d9b9-44a1-83b7-3d2b33f22dbe\") " Nov 01 05:43:44.303054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.303031 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca" pod "721b275d-d9b9-44a1-83b7-3d2b33f22dbe" (UID: "721b275d-d9b9-44a1-83b7-3d2b33f22dbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:44.303360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.303342 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:44.304978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.304956 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca" pod "721b275d-d9b9-44a1-83b7-3d2b33f22dbe" (UID: "721b275d-d9b9-44a1-83b7-3d2b33f22dbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:43:44.304978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.304970 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn" pod "721b275d-d9b9-44a1-83b7-3d2b33f22dbe" (UID: "721b275d-d9b9-44a1-83b7-3d2b33f22dbe"). InnerVolumeSpecName "kube-api-access-8q8pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:44.404818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.404788 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:44.404818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:44.404816 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8pn\" (UniqueName: \"kubernetes.io/projected/721b275d-d9b9-44a1-83b7-3d2b33f22dbe-kube-api-access-8q8pn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:45.032279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:45.032224 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" Nov 01 05:43:45.032470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:45.032305 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr" event={"ID":"721b275d-d9b9-44a1-83b7-3d2b33f22dbe","Type":"ContainerDied","Data":"ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403"} Nov 01 05:43:45.032470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:45.032373 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba35a23eca262b2416560acdbacd2ea383a12b428cbe7db1e443bf3580b4c403" Nov 01 05:43:45.114427 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.114384 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:45.114427 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.114425 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:45.114675 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.114500 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:43:45.614476461 +0000 UTC m=+2440.755065377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:45.621386 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.621345 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:45.621386 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.621382 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:45.621856 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:45.621453 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:43:46.621436141 +0000 UTC m=+2441.762025038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:46.634675 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:46.634640 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:46.634675 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:46.634672 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:46.635290 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:46.634755 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:43:48.634720603 +0000 UTC m=+2443.775309501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:48.664334 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:48.664289 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:48.664334 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:48.664332 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:48.664836 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:48.664392 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:43:52.664375644 +0000 UTC m=+2447.804964554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:48.967872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:48.967768 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:48.980002 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:48.979960 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:52.711050 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:52.711019 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:52.711050 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:52.711047 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:52.711525 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:52.711109 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:44:00.711091311 +0000 UTC m=+2455.851680225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:43:53.352972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.352927 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7"] Nov 01 05:43:53.353689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.353668 4425 cpu_manager.go:414] "containerMap: removing container" podUID="721b275d-d9b9-44a1-83b7-3d2b33f22dbe" containerName="revoke-keys" Nov 01 05:43:53.353803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.353690 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721b275d-d9b9-44a1-83b7-3d2b33f22dbe" containerName="revoke-keys" Nov 01 05:43:53.353889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.353870 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="721b275d-d9b9-44a1-83b7-3d2b33f22dbe" containerName="revoke-keys" Nov 01 05:43:53.357859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.357839 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.414773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.414741 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7"] Nov 01 05:43:53.419701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.419674 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.419867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.419712 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmxq\" (UniqueName: \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.419935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.419885 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.460671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.460633 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:43:53.460853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.460693 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:43:53.460853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.460705 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:43:53.460853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.460744 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:43:53.520835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.520799 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.521000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.520941 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.521000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.520970 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmxq\" (UniqueName: \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.521528 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.521510 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.523784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.523760 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.584915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.584881 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmxq\" (UniqueName: \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\") pod \"maas-api-revoke-sub-2285bbb6-rgzg7\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:53.671533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:53.671445 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:54.051202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:54.051171 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7"] Nov 01 05:43:54.051962 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:43:54.051930 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d1711d_a5ed_4208_a03e_575df3f53736.slice/crio-cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90 WatchSource:0}: Error finding container cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90: Status 404 returned error can't find the container with id cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90 Nov 01 05:43:54.094319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:54.094287 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" event={"ID":"64d1711d-a5ed-4208-a03e-575df3f53736","Type":"ContainerStarted","Data":"cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90"} Nov 01 05:43:55.102963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:55.102924 4425 generic.go:356] "Generic (PLEG): container finished" podID="64d1711d-a5ed-4208-a03e-575df3f53736" containerID="57aee33a83f15110f3c51f0a7d4280c52862d32e9bfca5e60193b39f25dcc80f" exitCode=0 Nov 01 05:43:55.103404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:55.103004 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" event={"ID":"64d1711d-a5ed-4208-a03e-575df3f53736","Type":"ContainerDied","Data":"57aee33a83f15110f3c51f0a7d4280c52862d32e9bfca5e60193b39f25dcc80f"} Nov 01 05:43:56.267591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.267563 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:56.351230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.351185 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\") pod \"64d1711d-a5ed-4208-a03e-575df3f53736\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " Nov 01 05:43:56.351459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.351318 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\") pod \"64d1711d-a5ed-4208-a03e-575df3f53736\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " Nov 01 05:43:56.351532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.351458 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\" (UniqueName: \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\") pod \"64d1711d-a5ed-4208-a03e-575df3f53736\" (UID: \"64d1711d-a5ed-4208-a03e-575df3f53736\") " Nov 01 05:43:56.351696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.351667 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca" pod "64d1711d-a5ed-4208-a03e-575df3f53736" (UID: "64d1711d-a5ed-4208-a03e-575df3f53736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:43:56.351960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.351940 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:56.353632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.353605 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca" pod "64d1711d-a5ed-4208-a03e-575df3f53736" (UID: "64d1711d-a5ed-4208-a03e-575df3f53736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:43:56.353765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.353658 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq" pod "64d1711d-a5ed-4208-a03e-575df3f53736" (UID: "64d1711d-a5ed-4208-a03e-575df3f53736"). InnerVolumeSpecName "kube-api-access-2cmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:43:56.453351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.453308 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d1711d-a5ed-4208-a03e-575df3f53736-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:56.453351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:56.453344 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmxq\" (UniqueName: \"kubernetes.io/projected/64d1711d-a5ed-4208-a03e-575df3f53736-kube-api-access-2cmxq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:43:56.947960 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:43:56.947916 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:43:57.117675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:57.117649 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" Nov 01 05:43:57.117842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:57.117686 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7" event={"ID":"64d1711d-a5ed-4208-a03e-575df3f53736","Type":"ContainerDied","Data":"cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90"} Nov 01 05:43:57.117842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:57.117733 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc35e59fbb7dcfb4ba8700cee8bb60c97f68a11f867cc8e84bf58ba82d272e90" Nov 01 05:43:58.967451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:58.967403 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:43:58.979300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:43:58.979247 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:00.711408 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:00.711379 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:00.711408 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:00.711405 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:00.711858 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:00.711464 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:44:16.711451354 +0000 UTC m=+2471.852040251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:05.090693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.090662 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6"] Nov 01 05:44:05.091283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.091268 4425 cpu_manager.go:414] "containerMap: removing container" podUID="64d1711d-a5ed-4208-a03e-575df3f53736" containerName="revoke-keys" Nov 01 05:44:05.091328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.091284 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d1711d-a5ed-4208-a03e-575df3f53736" containerName="revoke-keys" Nov 01 05:44:05.091442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.091431 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d1711d-a5ed-4208-a03e-575df3f53736" containerName="revoke-keys" Nov 01 05:44:05.095208 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.095182 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.153385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.153344 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6"] Nov 01 05:44:05.166409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.166365 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.166613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.166443 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrch\" (UniqueName: \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.166691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.166606 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.198619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.198586 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:44:05.198788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.198627 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:44:05.198788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.198651 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:44:05.199260 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.199239 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:44:05.268061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.268027 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.268295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.268109 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrch\" (UniqueName: \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.268295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.268156 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.268887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.268861 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.270593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.270575 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.329864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.329808 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrch\" (UniqueName: \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\") pod \"maas-api-revoke-sub-93010f46-5p6k6\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.409510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.409422 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:05.790327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:05.790302 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6"] Nov 01 05:44:05.792891 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:05.792861 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66751a77_c23f_40bc_bf6f_fb04056f5d28.slice/crio-e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d WatchSource:0}: Error finding container e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d: Status 404 returned error can't find the container with id e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d Nov 01 05:44:06.052078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.051930 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k"] Nov 01 05:44:06.055575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.055547 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad90c52d-qxs8k"] Nov 01 05:44:06.183822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.183782 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" event={"ID":"66751a77-c23f-40bc-bf6f-fb04056f5d28","Type":"ContainerStarted","Data":"143bbe3c740444ce1b7b199941dc04516dfa6e112cd6445d24bd0f989a54d970"} Nov 01 05:44:06.184287 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.183842 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" event={"ID":"66751a77-c23f-40bc-bf6f-fb04056f5d28","Type":"ContainerStarted","Data":"e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d"} Nov 01 05:44:06.300876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.300768 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" podStartSLOduration=1.300749358 podStartE2EDuration="1.300749358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:06.300416934 +0000 UTC m=+2461.441005856" watchObservedRunningTime="2026-11-01 05:44:06.300749358 +0000 UTC m=+2461.441338277" Nov 01 05:44:06.560442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.560362 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:44:06.560664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.560634 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" containerID="cri-o://9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7" gracePeriod=30 Nov 01 05:44:06.873943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.873910 4425 scope.go:122] "RemoveContainer" containerID="8907831176e3c2595617e98005f0a6ed42fb1fff4db88af5c3884e04e587b28c" Nov 01 05:44:06.974699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:06.974662 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf815a7-c470-428d-8b20-d7fdfcf98761" path="/var/lib/kubelet/pods/bcf815a7-c470-428d-8b20-d7fdfcf98761/volumes" Nov 01 05:44:07.192881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:07.192790 4425 generic.go:356] "Generic (PLEG): container finished" podID="66751a77-c23f-40bc-bf6f-fb04056f5d28" containerID="143bbe3c740444ce1b7b199941dc04516dfa6e112cd6445d24bd0f989a54d970" exitCode=0 Nov 01 05:44:07.193301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:07.192876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" event={"ID":"66751a77-c23f-40bc-bf6f-fb04056f5d28","Type":"ContainerDied","Data":"143bbe3c740444ce1b7b199941dc04516dfa6e112cd6445d24bd0f989a54d970"} Nov 01 05:44:08.368062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.367987 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:08.510287 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.510171 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\") pod \"66751a77-c23f-40bc-bf6f-fb04056f5d28\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " Nov 01 05:44:08.510538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.510395 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\" (UniqueName: \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\") pod \"66751a77-c23f-40bc-bf6f-fb04056f5d28\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " Nov 01 05:44:08.510538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.510508 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\") pod \"66751a77-c23f-40bc-bf6f-fb04056f5d28\" (UID: \"66751a77-c23f-40bc-bf6f-fb04056f5d28\") " Nov 01 05:44:08.510934 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.510908 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca" pod "66751a77-c23f-40bc-bf6f-fb04056f5d28" (UID: "66751a77-c23f-40bc-bf6f-fb04056f5d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:08.511137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.511118 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:08.512625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.512597 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca" pod "66751a77-c23f-40bc-bf6f-fb04056f5d28" (UID: "66751a77-c23f-40bc-bf6f-fb04056f5d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:44:08.513236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.513215 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch" pod "66751a77-c23f-40bc-bf6f-fb04056f5d28" (UID: "66751a77-c23f-40bc-bf6f-fb04056f5d28"). InnerVolumeSpecName "kube-api-access-scrch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:08.612549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.612518 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66751a77-c23f-40bc-bf6f-fb04056f5d28-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:08.612549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.612545 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrch\" (UniqueName: \"kubernetes.io/projected/66751a77-c23f-40bc-bf6f-fb04056f5d28-kube-api-access-scrch\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:08.967711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.967673 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:08.979433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:08.979398 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:09.207601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:09.207565 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" Nov 01 05:44:09.207771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:09.207609 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6" event={"ID":"66751a77-c23f-40bc-bf6f-fb04056f5d28","Type":"ContainerDied","Data":"e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d"} Nov 01 05:44:09.207771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:09.207656 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1bfc079558381348c0456173ab3c048c42a59a8daad73c94ffb8f4ba58e117d" Nov 01 05:44:11.832943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.832874 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:44:11.956186 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956153 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956227 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956284 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956383 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956449 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956515 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956552 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956582 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:44:11.956697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956611 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956641 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:44:11.956899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956719 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\") pod \"a6c211a5-586a-4f41-91b7-738cae284326\" (UID: \"a6c211a5-586a-4f41-91b7-738cae284326\") " Nov 01 05:44:11.956899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956796 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:44:11.956899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956819 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:11.957060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.956894 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:44:11.957447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.957315 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:11.957447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.957344 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:11.957447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.957359 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:11.957447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.957373 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c211a5-586a-4f41-91b7-738cae284326-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:11.957447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.957387 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:11.958881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.958858 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:44:11.958881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.958862 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:11.959117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.959098 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "kube-api-access-khhwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:11.959180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:11.959128 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy" pod "a6c211a5-586a-4f41-91b7-738cae284326" (UID: "a6c211a5-586a-4f41-91b7-738cae284326"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:44:12.058456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.058413 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhwp\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-kube-api-access-khhwp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:12.058456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.058446 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c211a5-586a-4f41-91b7-738cae284326-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:12.058456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.058462 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c211a5-586a-4f41-91b7-738cae284326-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:12.058750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.058474 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c211a5-586a-4f41-91b7-738cae284326-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:12.228988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.228894 4425 generic.go:356] "Generic (PLEG): container finished" podID="a6c211a5-586a-4f41-91b7-738cae284326" containerID="9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7" exitCode=0 Nov 01 05:44:12.228988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.228950 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" Nov 01 05:44:12.229216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.228981 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerDied","Data":"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7"} Nov 01 05:44:12.229216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.229053 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt" event={"ID":"a6c211a5-586a-4f41-91b7-738cae284326","Type":"ContainerDied","Data":"dde30f2c5283fbfffbc12e63f0c4b621c2f16268ffbde22063cb64b8a1ff6cc3"} Nov 01 05:44:12.229216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.229084 4425 scope.go:122] "RemoveContainer" containerID="9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7" Nov 01 05:44:12.240836 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.240813 4425 scope.go:122] "RemoveContainer" containerID="8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3" Nov 01 05:44:12.253069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.253037 4425 scope.go:122] "RemoveContainer" containerID="9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7" Nov 01 05:44:12.253412 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:12.253390 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7\": container with ID starting with 9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7 not found: ID does not exist" containerID="9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7" Nov 01 05:44:12.253493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.253421 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7"} err="failed to get container status \"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7\": rpc error: code = NotFound desc = could not find container \"9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7\": container with ID starting with 9b8391324fd16d060647140898075ab54c17d5e577e5dc5b03d4232eac813df7 not found: ID does not exist" Nov 01 05:44:12.253493 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.253441 4425 scope.go:122] "RemoveContainer" containerID="8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3" Nov 01 05:44:12.253721 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:12.253699 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3\": container with ID starting with 8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3 not found: ID does not exist" containerID="8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3" Nov 01 05:44:12.253781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.253732 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3"} err="failed to get container status \"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3\": rpc error: code = NotFound desc = could not find container \"8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3\": container with ID starting with 8ebe6a12b438efba4349ccb40f0d374abbc8b24168cff4301381c425f5aaf5f3 not found: ID does not exist" Nov 01 05:44:12.398195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.398159 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:44:12.402265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.402231 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad90c52d-openshift-default-6bc76dbd85-htpzt"] Nov 01 05:44:12.974266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:12.974221 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c211a5-586a-4f41-91b7-738cae284326" path="/var/lib/kubelet/pods/a6c211a5-586a-4f41-91b7-738cae284326/volumes" Nov 01 05:44:15.122486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.122436 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:44:15.124140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124111 4425 cpu_manager.go:414] "containerMap: removing container" podUID="66751a77-c23f-40bc-bf6f-fb04056f5d28" containerName="revoke-keys" Nov 01 05:44:15.124335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124320 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66751a77-c23f-40bc-bf6f-fb04056f5d28" containerName="revoke-keys" Nov 01 05:44:15.124460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124450 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.124538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124527 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.124639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124629 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.124706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.124697 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.127087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.127067 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.127278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.127245 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="66751a77-c23f-40bc-bf6f-fb04056f5d28" containerName="revoke-keys" Nov 01 05:44:15.136051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.135380 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.176124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.176078 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:44:15.197654 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.197622 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.197862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.197663 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flzn\" (UniqueName: \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.221944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.221885 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-31e728ff\"" Nov 01 05:44:15.222142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.222122 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-31e728ff-dockercfg-qznm5\"" Nov 01 05:44:15.293033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.292997 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:44:15.293754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.293739 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c211a5-586a-4f41-91b7-738cae284326" containerName="istio-proxy" Nov 01 05:44:15.297301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.297283 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.298747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.298722 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.298879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.298760 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flzn\" (UniqueName: \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.299505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.299476 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.356628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.356591 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:44:15.361860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.361826 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flzn\" (UniqueName: \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\") pod \"payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.400004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.399914 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2js\" (UniqueName: \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.400004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.399968 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.461729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.461690 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:15.500958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.500919 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2js\" (UniqueName: \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.501180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.500987 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.502028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.502002 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.565859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.565818 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2js\" (UniqueName: \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\") pod \"payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.612081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.612044 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:15.847945 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:15.847902 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb966c01f_27e5_4640_b691_a2a43e52a380.slice/crio-3bfa1eefc7a4ca0e1e686f2713a89e94ae0c98942f8dd91934717574aab714b2 WatchSource:0}: Error finding container 3bfa1eefc7a4ca0e1e686f2713a89e94ae0c98942f8dd91934717574aab714b2: Status 404 returned error can't find the container with id 3bfa1eefc7a4ca0e1e686f2713a89e94ae0c98942f8dd91934717574aab714b2 Nov 01 05:44:15.848113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.848021 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:44:15.996427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:15.996395 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:44:15.997620 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:15.997592 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff62ef87_25eb_419c_a142_ccae8827781b.slice/crio-699a347276769e560d977fa5a0145edce78a37df50eff38e49dcfbb08e233b61 WatchSource:0}: Error finding container 699a347276769e560d977fa5a0145edce78a37df50eff38e49dcfbb08e233b61: Status 404 returned error can't find the container with id 699a347276769e560d977fa5a0145edce78a37df50eff38e49dcfbb08e233b61 Nov 01 05:44:16.258190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.258092 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" event={"ID":"b966c01f-27e5-4640-b691-a2a43e52a380","Type":"ContainerStarted","Data":"238bbbc9e1cdfeb4b8a73efd15cd19062c8fb12dc34c261e98360a82a537ecc8"} Nov 01 05:44:16.258190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.258147 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" event={"ID":"b966c01f-27e5-4640-b691-a2a43e52a380","Type":"ContainerStarted","Data":"3bfa1eefc7a4ca0e1e686f2713a89e94ae0c98942f8dd91934717574aab714b2"} Nov 01 05:44:16.258190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.258176 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:16.259973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.259944 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" event={"ID":"ff62ef87-25eb-419c-a142-ccae8827781b","Type":"ContainerStarted","Data":"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8"} Nov 01 05:44:16.260101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.259995 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" event={"ID":"ff62ef87-25eb-419c-a142-ccae8827781b","Type":"ContainerStarted","Data":"699a347276769e560d977fa5a0145edce78a37df50eff38e49dcfbb08e233b61"} Nov 01 05:44:16.260101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.260016 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:16.378190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:16.378094 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" podStartSLOduration=1.378079061 podStartE2EDuration="1.378079061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:16.375551452 +0000 UTC m=+2471.516140374" watchObservedRunningTime="2026-11-01 05:44:16.378079061 +0000 UTC m=+2471.518668024" Nov 01 05:44:16.717366 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:16.717326 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:16.717366 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:16.717365 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:16.717614 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:16.717436 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:44:48.71741508 +0000 UTC m=+2503.858003998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:18.967782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:18.967743 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:18.978615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:18.978580 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:20.433397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.433339 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" podStartSLOduration=5.433324488 podStartE2EDuration="5.433324488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:16.494486492 +0000 UTC m=+2471.635075431" watchObservedRunningTime="2026-11-01 05:44:20.433324488 +0000 UTC m=+2475.573913431" Nov 01 05:44:20.434240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.434223 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v"] Nov 01 05:44:20.438831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.438814 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.495443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.495410 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v"] Nov 01 05:44:20.541964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.541922 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:44:20.542141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.541990 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:44:20.542141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.541999 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:44:20.542495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.542479 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:44:20.567581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.567545 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.567763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.567703 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.567880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.567867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwh7c\" (UniqueName: \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.668830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.668802 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwh7c\" (UniqueName: \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.669029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.668851 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.669029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.668997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.669671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.669646 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.671331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.671311 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.730940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.730860 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwh7c\" (UniqueName: \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\") pod \"maas-api-revoke-sub-707d87a7-bww8v\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:20.754463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:20.754428 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:21.133653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:21.133622 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v"] Nov 01 05:44:21.134348 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:21.134323 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1c71cb_9479_42c9_98e7_b476be5a2035.slice/crio-3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7 WatchSource:0}: Error finding container 3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7: Status 404 returned error can't find the container with id 3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7 Nov 01 05:44:21.302316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:21.302276 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" event={"ID":"3f1c71cb-9479-42c9-98e7-b476be5a2035","Type":"ContainerStarted","Data":"3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7"} Nov 01 05:44:22.310411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:22.310373 4425 generic.go:356] "Generic (PLEG): container finished" podID="3f1c71cb-9479-42c9-98e7-b476be5a2035" containerID="cfa1ba5d1c9d51a0199ce4a98c93b34de0b3e2f0c4422a0d8887e2dcd3222ed7" exitCode=0 Nov 01 05:44:22.310813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:22.310419 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" event={"ID":"3f1c71cb-9479-42c9-98e7-b476be5a2035","Type":"ContainerDied","Data":"cfa1ba5d1c9d51a0199ce4a98c93b34de0b3e2f0c4422a0d8887e2dcd3222ed7"} Nov 01 05:44:23.483904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.483874 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:23.605821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.605730 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\" (UniqueName: \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\") pod \"3f1c71cb-9479-42c9-98e7-b476be5a2035\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " Nov 01 05:44:23.606004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.605840 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\") pod \"3f1c71cb-9479-42c9-98e7-b476be5a2035\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " Nov 01 05:44:23.606004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.605897 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\") pod \"3f1c71cb-9479-42c9-98e7-b476be5a2035\" (UID: \"3f1c71cb-9479-42c9-98e7-b476be5a2035\") " Nov 01 05:44:23.606399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.606368 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca" pod "3f1c71cb-9479-42c9-98e7-b476be5a2035" (UID: "3f1c71cb-9479-42c9-98e7-b476be5a2035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:23.606566 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.606548 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:23.608844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.608817 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca" pod "3f1c71cb-9479-42c9-98e7-b476be5a2035" (UID: "3f1c71cb-9479-42c9-98e7-b476be5a2035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:44:23.609080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.609045 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c" pod "3f1c71cb-9479-42c9-98e7-b476be5a2035" (UID: "3f1c71cb-9479-42c9-98e7-b476be5a2035"). InnerVolumeSpecName "kube-api-access-wwh7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:23.707701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.707663 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwh7c\" (UniqueName: \"kubernetes.io/projected/3f1c71cb-9479-42c9-98e7-b476be5a2035-kube-api-access-wwh7c\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:23.707701 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:23.707701 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c71cb-9479-42c9-98e7-b476be5a2035-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:24.325414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:24.325383 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" Nov 01 05:44:24.325574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:24.325383 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v" event={"ID":"3f1c71cb-9479-42c9-98e7-b476be5a2035","Type":"ContainerDied","Data":"3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7"} Nov 01 05:44:24.325574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:24.325512 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3459f9e648665d3a1e2d48078a7cfb4be8a7173591f17ef6456b912f153181f7" Nov 01 05:44:26.949426 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:26.949383 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:44:27.270973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:27.270876 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:44:27.270973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:27.270969 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:44:28.968132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:28.968082 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:28.978542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:28.978508 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:31.572081 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:31.572040 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" podUID="dcfcbe84-496c-46b4-a708-aec6bde4700e" Nov 01 05:44:33.530546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.530510 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp"] Nov 01 05:44:33.531362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.531341 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3f1c71cb-9479-42c9-98e7-b476be5a2035" containerName="revoke-keys" Nov 01 05:44:33.531492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.531363 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1c71cb-9479-42c9-98e7-b476be5a2035" containerName="revoke-keys" Nov 01 05:44:33.531590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.531576 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1c71cb-9479-42c9-98e7-b476be5a2035" containerName="revoke-keys" Nov 01 05:44:33.535141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.535118 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.588796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.588748 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:44:33.592127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.592098 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp"] Nov 01 05:44:33.634881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.634836 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.635101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.634948 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.635101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.635092 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtd7b\" (UniqueName: \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.638611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.638564 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:44:33.638611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.638606 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:44:33.638809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.638673 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:44:33.736890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.736853 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.737122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.736951 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.737187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.737152 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtd7b\" (UniqueName: \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.738024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.737988 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.739795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.739774 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.799449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.799367 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtd7b\" (UniqueName: \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\") pod \"maas-api-revoke-sub-33eb21d5-7p2pp\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:33.852992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:33.852955 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:34.239618 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:34.239587 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp"] Nov 01 05:44:34.240013 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:34.239984 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5a6546_3811_45af_8afc_1f4cfbd9ff4d.slice/crio-3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb WatchSource:0}: Error finding container 3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb: Status 404 returned error can't find the container with id 3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb Nov 01 05:44:34.395857 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:34.395817 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" event={"ID":"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d","Type":"ContainerStarted","Data":"3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb"} Nov 01 05:44:35.245338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.245309 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:44:35.247052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.247022 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:35.295696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.295656 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:35.300010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.299992 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.357793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.357760 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:35.404209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.404175 4425 generic.go:356] "Generic (PLEG): container finished" podID="1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" containerID="8e9526e4ed716d37031ff8475cf3401093352f01e96cc8d741283ca0b537bcb1" exitCode=0 Nov 01 05:44:35.404454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.404318 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" event={"ID":"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d","Type":"ContainerDied","Data":"8e9526e4ed716d37031ff8475cf3401093352f01e96cc8d741283ca0b537bcb1"} Nov 01 05:44:35.458781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.458681 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68klw\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.458948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.458834 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.459009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.458982 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.459055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.459039 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.560271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.560224 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.560486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.560437 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68klw\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.560565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.560498 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.560635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.560617 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.560983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.560955 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.561364 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.561339 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.563351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.563327 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.624222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.624187 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68klw\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:35.638070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:35.638045 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:36.030424 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:36.030378 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef8ca3c_2fed_446c_93cc_b80346d64730.slice/crio-434c03b30f3562da0210dc4dfec110822c828f1ebb92eb3930e8baf1d5e6bdba WatchSource:0}: Error finding container 434c03b30f3562da0210dc4dfec110822c828f1ebb92eb3930e8baf1d5e6bdba: Status 404 returned error can't find the container with id 434c03b30f3562da0210dc4dfec110822c828f1ebb92eb3930e8baf1d5e6bdba Nov 01 05:44:36.030997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.030971 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:36.047549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.047517 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4"] Nov 01 05:44:36.051001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.050978 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a50973c-pcbc4"] Nov 01 05:44:36.412843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.412808 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" event={"ID":"6ef8ca3c-2fed-446c-93cc-b80346d64730","Type":"ContainerStarted","Data":"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a"} Nov 01 05:44:36.413334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.412864 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" event={"ID":"6ef8ca3c-2fed-446c-93cc-b80346d64730","Type":"ContainerStarted","Data":"434c03b30f3562da0210dc4dfec110822c828f1ebb92eb3930e8baf1d5e6bdba"} Nov 01 05:44:36.413334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.412951 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:36.472275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.472221 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:44:36.472545 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:36.472521 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:44:36.472629 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:36.472548 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh: secret "agc-cred-dc7d983de1b3" not found Nov 01 05:44:36.472629 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:36.472612 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials podName:dcfcbe84-496c-46b4-a708-aec6bde4700e nodeName:}" failed. No retries permitted until 2026-11-01 05:46:38.472592961 +0000 UTC m=+2613.613181872 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e") : secret "agc-cred-dc7d983de1b3" not found Nov 01 05:44:36.535268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.535194 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" podStartSLOduration=1.535177065 podStartE2EDuration="1.535177065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:36.531985025 +0000 UTC m=+2491.672573945" watchObservedRunningTime="2026-11-01 05:44:36.535177065 +0000 UTC m=+2491.675766011" Nov 01 05:44:36.566815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.566784 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:36.674916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.674827 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\") pod \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " Nov 01 05:44:36.674916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.674902 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\" (UniqueName: \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\") pod \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " Nov 01 05:44:36.675140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.675041 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\") pod \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\" (UID: \"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d\") " Nov 01 05:44:36.675407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.675384 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca" pod "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" (UID: "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:36.675560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.675526 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:36.677198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.677178 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca" pod "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" (UID: "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:44:36.677276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.677203 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b" pod "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" (UID: "1c5a6546-3811-45af-8afc-1f4cfbd9ff4d"). InnerVolumeSpecName "kube-api-access-wtd7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:36.776690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.776658 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:36.776690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.776687 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtd7b\" (UniqueName: \"kubernetes.io/projected/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d-kube-api-access-wtd7b\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:36.974750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:36.974674 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5368d7e-9aaf-4e5e-8429-299d89c55c86" path="/var/lib/kubelet/pods/b5368d7e-9aaf-4e5e-8429-299d89c55c86/volumes" Nov 01 05:44:37.421572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:37.421538 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" Nov 01 05:44:37.421994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:37.421547 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp" event={"ID":"1c5a6546-3811-45af-8afc-1f4cfbd9ff4d","Type":"ContainerDied","Data":"3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb"} Nov 01 05:44:37.421994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:37.421664 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3843b30f390d76f1015a3b6c05385a58c044c28434ca496447ecdd684afeecbb" Nov 01 05:44:38.967505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:38.967463 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:40.246022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.245977 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" containerID="cri-o://14e403c870a1e94b10d2bd6f5e4ff537460bc154955fd13f8929e590fad0cbe8" gracePeriod=25 Nov 01 05:44:40.445197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.445159 4425 generic.go:356] "Generic (PLEG): container finished" podID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerID="14e403c870a1e94b10d2bd6f5e4ff537460bc154955fd13f8929e590fad0cbe8" exitCode=0 Nov 01 05:44:40.445412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.445291 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" event={"ID":"c89ddc5e-fa32-4226-9f97-e446c54aae84","Type":"ContainerDied","Data":"14e403c870a1e94b10d2bd6f5e4ff537460bc154955fd13f8929e590fad0cbe8"} Nov 01 05:44:40.499874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.499810 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:44:40.525979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.525952 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\") pod \"c89ddc5e-fa32-4226-9f97-e446c54aae84\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " Nov 01 05:44:40.526126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526036 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\") pod \"c89ddc5e-fa32-4226-9f97-e446c54aae84\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " Nov 01 05:44:40.526126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526105 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\") pod \"c89ddc5e-fa32-4226-9f97-e446c54aae84\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " Nov 01 05:44:40.526282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526225 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\") pod \"c89ddc5e-fa32-4226-9f97-e446c54aae84\" (UID: \"c89ddc5e-fa32-4226-9f97-e446c54aae84\") " Nov 01 05:44:40.526575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526514 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume" pod "c89ddc5e-fa32-4226-9f97-e446c54aae84" (UID: "c89ddc5e-fa32-4226-9f97-e446c54aae84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:40.526672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526619 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume" pod "c89ddc5e-fa32-4226-9f97-e446c54aae84" (UID: "c89ddc5e-fa32-4226-9f97-e446c54aae84"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:40.526843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526826 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:40.526919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.526849 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c89ddc5e-fa32-4226-9f97-e446c54aae84-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:40.528487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.528460 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials" pod "c89ddc5e-fa32-4226-9f97-e446c54aae84" (UID: "c89ddc5e-fa32-4226-9f97-e446c54aae84"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:40.528576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.528492 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw" pod "c89ddc5e-fa32-4226-9f97-e446c54aae84" (UID: "c89ddc5e-fa32-4226-9f97-e446c54aae84"). InnerVolumeSpecName "kube-api-access-cmmqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:40.628053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.628021 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:40.628053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:40.628050 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmqw\" (UniqueName: \"kubernetes.io/projected/c89ddc5e-fa32-4226-9f97-e446c54aae84-kube-api-access-cmmqw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:41.453443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:41.453406 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" event={"ID":"c89ddc5e-fa32-4226-9f97-e446c54aae84","Type":"ContainerDied","Data":"5250d523e17a29647f8bf95fd1fc04cadb23c5fa04e7f1182a8c3ee1d049e12e"} Nov 01 05:44:41.453895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:41.453468 4425 scope.go:122] "RemoveContainer" containerID="14e403c870a1e94b10d2bd6f5e4ff537460bc154955fd13f8929e590fad0cbe8" Nov 01 05:44:41.453895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:41.453424 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm" Nov 01 05:44:41.625829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:41.625799 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:44:41.629104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:41.629081 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6b9dsm"] Nov 01 05:44:42.965877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:42.965844 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:44:42.973802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:42.973771 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" path="/var/lib/kubelet/pods/c89ddc5e-fa32-4226-9f97-e446c54aae84/volumes" Nov 01 05:44:46.673345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.673311 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp"] Nov 01 05:44:46.673961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.673945 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" Nov 01 05:44:46.674008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.673961 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" Nov 01 05:44:46.674008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.674002 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" containerName="revoke-keys" Nov 01 05:44:46.674079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.674010 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" containerName="revoke-keys" Nov 01 05:44:46.674116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.674104 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" containerName="revoke-keys" Nov 01 05:44:46.674156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.674116 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89ddc5e-fa32-4226-9f97-e446c54aae84" containerName="payload-processing" Nov 01 05:44:46.679297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.679247 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.736444 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.736412 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp"] Nov 01 05:44:46.781367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.781334 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:44:46.781557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.781395 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:44:46.783456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.783439 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:44:46.783725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.783711 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:44:46.811603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.811574 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42r8\" (UniqueName: \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.811774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.811640 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.811840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.811819 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.913125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.913088 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.913347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.913227 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42r8\" (UniqueName: \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.913347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.913298 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.914025 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.913998 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.915728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.915710 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:46.975581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:46.975513 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42r8\" (UniqueName: \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\") pod \"maas-api-revoke-sub-a5c5d33c-mp6kp\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:47.000752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.000462 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:47.033279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.033226 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:47.036457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.036425 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:47.060919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.060877 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:44:47.074826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.074793 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5"] Nov 01 05:44:47.074955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.074841 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfd929-l6xw5"] Nov 01 05:44:47.074955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.074938 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.126445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.121376 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:44:47.216555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.216525 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwcs\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.216743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.216661 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.216743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.216692 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.216824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.216802 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.319028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.318978 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwcs\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.319234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.319097 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.319234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.319127 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.319234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.319203 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.320415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.320385 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.321092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.321066 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.323936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.323911 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.383229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.383184 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwcs\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.391580 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:47.391552 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b6083a_a488_4112_b9b7_37741243d7f6.slice/crio-4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b WatchSource:0}: Error finding container 4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b: Status 404 returned error can't find the container with id 4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b Nov 01 05:44:47.391778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.391756 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp"] Nov 01 05:44:47.397935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.397913 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:47.500796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.500757 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" event={"ID":"15b6083a-a488-4112-b9b7-37741243d7f6","Type":"ContainerStarted","Data":"4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b"} Nov 01 05:44:47.780486 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:44:47.780456 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964382ee_4646_4bc0_90fa_e13cf4d1d42a.slice/crio-eea27c9870bae52edc3b2968bc16aa836bb518297f4d40d38e628f61d8e39873 WatchSource:0}: Error finding container eea27c9870bae52edc3b2968bc16aa836bb518297f4d40d38e628f61d8e39873: Status 404 returned error can't find the container with id eea27c9870bae52edc3b2968bc16aa836bb518297f4d40d38e628f61d8e39873 Nov 01 05:44:47.781196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:47.781125 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:44:48.521028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.520911 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" event={"ID":"964382ee-4646-4bc0-90fa-e13cf4d1d42a","Type":"ContainerStarted","Data":"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63"} Nov 01 05:44:48.521028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.521014 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" event={"ID":"964382ee-4646-4bc0-90fa-e13cf4d1d42a","Type":"ContainerStarted","Data":"eea27c9870bae52edc3b2968bc16aa836bb518297f4d40d38e628f61d8e39873"} Nov 01 05:44:48.521342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.521081 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:44:48.522976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.522952 4425 generic.go:356] "Generic (PLEG): container finished" podID="15b6083a-a488-4112-b9b7-37741243d7f6" containerID="3b0dbdbd22d858cf5bbd25f391c977770837b17f505541d333829a78c82384b1" exitCode=0 Nov 01 05:44:48.523112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.523007 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" event={"ID":"15b6083a-a488-4112-b9b7-37741243d7f6","Type":"ContainerDied","Data":"3b0dbdbd22d858cf5bbd25f391c977770837b17f505541d333829a78c82384b1"} Nov 01 05:44:48.642515 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.642456 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" podStartSLOduration=1.642442195 podStartE2EDuration="1.642442195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:48.639338449 +0000 UTC m=+2503.779927369" watchObservedRunningTime="2026-11-01 05:44:48.642442195 +0000 UTC m=+2503.783031113" Nov 01 05:44:48.738089 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:48.738052 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-17306a2b8c16: secret "agc-cred-17306a2b8c16" not found Nov 01 05:44:48.738089 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:48.738082 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f16ea820c5f3: secret "agc-cred-f16ea820c5f3" not found Nov 01 05:44:48.738089 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:48.738089 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2fd4c9/agc-cred-f8c698a71031: secret "agc-cred-f8c698a71031" not found Nov 01 05:44:48.738089 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:48.738098 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf: [secret "agc-cred-17306a2b8c16" not found, secret "agc-cred-f16ea820c5f3" not found, secret "agc-cred-f8c698a71031" not found] Nov 01 05:44:48.738447 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:48.738163 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials podName:4e7d9053-976a-4578-b5c5-470fa5966aea nodeName:}" failed. No retries permitted until 2026-11-01 05:45:52.738146087 +0000 UTC m=+2567.878734984 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea") : [secret "agc-cred-17306a2b8c16" not found, secret "agc-cred-f16ea820c5f3" not found, secret "agc-cred-f8c698a71031" not found] Nov 01 05:44:48.968079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.968038 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:48.974745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:48.974708 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688126f6-7937-42d4-9c4d-3f408b0ee0ea" path="/var/lib/kubelet/pods/688126f6-7937-42d4-9c4d-3f408b0ee0ea/volumes" Nov 01 05:44:49.691023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.690994 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:49.750419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.750384 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\") pod \"15b6083a-a488-4112-b9b7-37741243d7f6\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " Nov 01 05:44:49.750592 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.750577 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\") pod \"15b6083a-a488-4112-b9b7-37741243d7f6\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " Nov 01 05:44:49.750756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.750660 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\" (UniqueName: \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\") pod \"15b6083a-a488-4112-b9b7-37741243d7f6\" (UID: \"15b6083a-a488-4112-b9b7-37741243d7f6\") " Nov 01 05:44:49.750756 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.750740 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca" pod "15b6083a-a488-4112-b9b7-37741243d7f6" (UID: "15b6083a-a488-4112-b9b7-37741243d7f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:49.751104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.751078 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:49.752956 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.752934 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca" pod "15b6083a-a488-4112-b9b7-37741243d7f6" (UID: "15b6083a-a488-4112-b9b7-37741243d7f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:44:49.753076 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.753043 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8" pod "15b6083a-a488-4112-b9b7-37741243d7f6" (UID: "15b6083a-a488-4112-b9b7-37741243d7f6"). InnerVolumeSpecName "kube-api-access-g42r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:49.852527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.852484 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g42r8\" (UniqueName: \"kubernetes.io/projected/15b6083a-a488-4112-b9b7-37741243d7f6-kube-api-access-g42r8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:49.852527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:49.852511 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b6083a-a488-4112-b9b7-37741243d7f6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:50.539779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:50.539740 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" Nov 01 05:44:50.539779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:50.539766 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp" event={"ID":"15b6083a-a488-4112-b9b7-37741243d7f6","Type":"ContainerDied","Data":"4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b"} Nov 01 05:44:50.540217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:50.539825 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed571dfc883b2d32070107948a174deca24e82a6a6b4095fba91e2c4b4c959b" Nov 01 05:44:52.034183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.034135 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerName="payload-processing" containerID="cri-o://ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a" gracePeriod=25 Nov 01 05:44:52.311911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.311276 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:52.395017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.394981 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\") pod \"6ef8ca3c-2fed-446c-93cc-b80346d64730\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " Nov 01 05:44:52.395205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.395098 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\") pod \"6ef8ca3c-2fed-446c-93cc-b80346d64730\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " Nov 01 05:44:52.395205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.395124 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\") pod \"6ef8ca3c-2fed-446c-93cc-b80346d64730\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " Nov 01 05:44:52.395205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.395162 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\") pod \"6ef8ca3c-2fed-446c-93cc-b80346d64730\" (UID: \"6ef8ca3c-2fed-446c-93cc-b80346d64730\") " Nov 01 05:44:52.395550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.395524 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume" pod "6ef8ca3c-2fed-446c-93cc-b80346d64730" (UID: "6ef8ca3c-2fed-446c-93cc-b80346d64730"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:52.395610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.395581 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume" pod "6ef8ca3c-2fed-446c-93cc-b80346d64730" (UID: "6ef8ca3c-2fed-446c-93cc-b80346d64730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:44:52.397466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.397443 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw" pod "6ef8ca3c-2fed-446c-93cc-b80346d64730" (UID: "6ef8ca3c-2fed-446c-93cc-b80346d64730"). InnerVolumeSpecName "kube-api-access-68klw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:52.397599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.397580 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials" pod "6ef8ca3c-2fed-446c-93cc-b80346d64730" (UID: "6ef8ca3c-2fed-446c-93cc-b80346d64730"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:44:52.497394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.497343 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:52.497623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.497606 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68klw\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-kube-api-access-68klw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:52.497766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.497753 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ef8ca3c-2fed-446c-93cc-b80346d64730-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:52.497857 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.497844 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ef8ca3c-2fed-446c-93cc-b80346d64730-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:44:52.566482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.566380 4425 generic.go:356] "Generic (PLEG): container finished" podID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerID="ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a" exitCode=0 Nov 01 05:44:52.569436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.566508 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" Nov 01 05:44:52.569436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.566531 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" event={"ID":"6ef8ca3c-2fed-446c-93cc-b80346d64730","Type":"ContainerDied","Data":"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a"} Nov 01 05:44:52.569436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.567654 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt" event={"ID":"6ef8ca3c-2fed-446c-93cc-b80346d64730","Type":"ContainerDied","Data":"434c03b30f3562da0210dc4dfec110822c828f1ebb92eb3930e8baf1d5e6bdba"} Nov 01 05:44:52.569436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.567710 4425 scope.go:122] "RemoveContainer" containerID="ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a" Nov 01 05:44:52.581416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.581392 4425 scope.go:122] "RemoveContainer" containerID="ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a" Nov 01 05:44:52.581756 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:52.581733 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a\": container with ID starting with ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a not found: ID does not exist" containerID="ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a" Nov 01 05:44:52.581834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.581766 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a"} err="failed to get container status \"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a\": rpc error: code = NotFound desc = could not find container \"ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a\": container with ID starting with ca756bea1880dee40573b123e2704d5688311d307ef979369365f09d66ea834a not found: ID does not exist" Nov 01 05:44:52.738395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.738352 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:52.770466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.770425 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9fd7rprlt"] Nov 01 05:44:52.976178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:52.976099 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" path="/var/lib/kubelet/pods/6ef8ca3c-2fed-446c-93cc-b80346d64730/volumes" Nov 01 05:44:56.973797 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:44:56.973744 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:44:58.967533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:58.967495 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:58.968629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:58.968592 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:44:59.074690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:59.074644 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:44:59.083931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:59.083889 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:44:59.087141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:44:59.087103 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:00.169396 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.169349 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:00.170371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170346 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerName="payload-processing" Nov 01 05:45:00.170371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170369 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerName="payload-processing" Nov 01 05:45:00.170578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170437 4425 cpu_manager.go:414] "containerMap: removing container" podUID="15b6083a-a488-4112-b9b7-37741243d7f6" containerName="revoke-keys" Nov 01 05:45:00.170578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170445 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b6083a-a488-4112-b9b7-37741243d7f6" containerName="revoke-keys" Nov 01 05:45:00.170698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170597 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef8ca3c-2fed-446c-93cc-b80346d64730" containerName="payload-processing" Nov 01 05:45:00.170698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.170618 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b6083a-a488-4112-b9b7-37741243d7f6" containerName="revoke-keys" Nov 01 05:45:00.174903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.174875 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:00.175055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.174988 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.179942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.179914 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:00.180093 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.180034 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.184154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.184129 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.220276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.220222 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:00.232276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.232220 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:00.233745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.233716 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235010 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fjw\" (UniqueName: \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235063 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235200 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx86p\" (UniqueName: \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235317 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235413 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.236272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.235688 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fwm\" (UniqueName: \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.239428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.239401 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:00.239551 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.239520 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.244373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.244350 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:00.244488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.244459 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.258475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.258444 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:00.258625 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.258598 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.263215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.263191 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:00.263376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.263327 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.267732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.267361 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:45:00.267732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.267385 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:45:00.267732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.267405 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.267732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.267476 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:45:00.273911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.273884 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:00.278894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.278869 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:00.282810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.282789 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:00.286232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.286203 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:00.333580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.333542 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:00.337410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337372 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.337574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpvj\" (UniqueName: \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.337574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337475 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.337574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337528 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kxc\" (UniqueName: \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.337574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337560 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2sb\" (UniqueName: \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.337770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337595 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.337770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337659 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.337770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337745 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.337871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337791 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf8j\" (UniqueName: \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.337871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337848 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fwm\" (UniqueName: \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.337943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337884 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fjw\" (UniqueName: \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.337943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337915 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.338010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337944 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlxx\" (UniqueName: \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.338010 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.337991 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.338075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.338029 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.338075 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.338061 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.338167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.338146 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46xf\" (UniqueName: \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.338297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.338235 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx86p\" (UniqueName: \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.340013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.339968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.340877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.340848 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.342238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.342210 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.342411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.342368 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:00.345079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.345043 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:00.377616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.377576 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:00.387092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.387054 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:00.403324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.403291 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx86p\" (UniqueName: \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\") pod \"maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.404183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.404126 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fwm\" (UniqueName: \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.404346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.404231 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fjw\" (UniqueName: \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.440027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.439942 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.440027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.439990 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpvj\" (UniqueName: \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.440316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440027 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.440316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440073 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kxc\" (UniqueName: \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.440316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440102 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2sb\" (UniqueName: \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.440316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440217 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.440316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440295 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf8j\" (UniqueName: \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.440601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440363 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlxx\" (UniqueName: \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.440601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440406 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.440601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440443 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.440601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440474 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.440601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440549 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46xf\" (UniqueName: \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.440884 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440862 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.440946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.440924 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.441286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.441234 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.441382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.441301 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.441599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.441576 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.441637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.441576 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.492078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.492041 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:00.499683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.499657 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:00.503280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.503219 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2sb\" (UniqueName: \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\") pod \"maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.503413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.503219 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpvj\" (UniqueName: \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.504019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.503980 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf8j\" (UniqueName: \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\") pod \"maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.504078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.504044 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46xf\" (UniqueName: \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.505309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.505285 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlxx\" (UniqueName: \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\") pod \"maas-api-key-cleanup-29891865-xgs2q\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.511903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.511873 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kxc\" (UniqueName: \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.513327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.513298 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:00.553455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.553418 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:00.566802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.566766 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:00.581090 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.578707 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:00.596503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.596475 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:00.597487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.597462 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:00.608784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.608751 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:00.895160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.895131 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:00.907972 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:00.907937 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8356e552_9856_407e_b5ee_c4178e730b67.slice/crio-59d09e9429a33c90967df546b7196882747ab8b8c91989b9f826c62ab6fa23a1 WatchSource:0}: Error finding container 59d09e9429a33c90967df546b7196882747ab8b8c91989b9f826c62ab6fa23a1: Status 404 returned error can't find the container with id 59d09e9429a33c90967df546b7196882747ab8b8c91989b9f826c62ab6fa23a1 Nov 01 05:45:00.945111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:00.945074 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:01.016902 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.016867 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:01.037341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.037310 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:01.044846 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:01.044780 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a0be2c_7ffc_4892_9b04_c7454b8a780e.slice/crio-78e81a5060b080ccdec70b9b695780035882bec61aba12852a6415da1dc43741 WatchSource:0}: Error finding container 78e81a5060b080ccdec70b9b695780035882bec61aba12852a6415da1dc43741: Status 404 returned error can't find the container with id 78e81a5060b080ccdec70b9b695780035882bec61aba12852a6415da1dc43741 Nov 01 05:45:01.046314 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:01.046286 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb12804_4ab5_4418_b081_cbd6a5708133.slice/crio-f6941642d6ed89c40678133bab2c035fc3132dbf6a942f9961ef41f7fffc240a WatchSource:0}: Error finding container f6941642d6ed89c40678133bab2c035fc3132dbf6a942f9961ef41f7fffc240a: Status 404 returned error can't find the container with id f6941642d6ed89c40678133bab2c035fc3132dbf6a942f9961ef41f7fffc240a Nov 01 05:45:01.073410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.073381 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:01.155947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.155791 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:01.160841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.160815 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:01.164678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.164651 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:01.316019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.315981 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:01.316468 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:01.316352 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3069f400_066e_4467_8855_6f97713c7470.slice/crio-cc60f64ef45acff6183612e3d8a01878d64b49e10bc0c4af8373a3af754ce2ee WatchSource:0}: Error finding container cc60f64ef45acff6183612e3d8a01878d64b49e10bc0c4af8373a3af754ce2ee: Status 404 returned error can't find the container with id cc60f64ef45acff6183612e3d8a01878d64b49e10bc0c4af8373a3af754ce2ee Nov 01 05:45:01.646595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.646498 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerStarted","Data":"cc60f64ef45acff6183612e3d8a01878d64b49e10bc0c4af8373a3af754ce2ee"} Nov 01 05:45:01.648712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.648674 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerStarted","Data":"f6941642d6ed89c40678133bab2c035fc3132dbf6a942f9961ef41f7fffc240a"} Nov 01 05:45:01.651156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.651121 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerStarted","Data":"78e81a5060b080ccdec70b9b695780035882bec61aba12852a6415da1dc43741"} Nov 01 05:45:01.653729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.653407 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerStarted","Data":"f19a6bdce111e4b663940377d002a262286108cdc722288895cbb9bf5d2f5fbd"} Nov 01 05:45:01.653729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.653476 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerStarted","Data":"59d09e9429a33c90967df546b7196882747ab8b8c91989b9f826c62ab6fa23a1"} Nov 01 05:45:01.655751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.655724 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerStarted","Data":"49ad5fa7fe34ca5e8778e1362566b5622681d8de1e1e4ec070f9c343a612aad5"} Nov 01 05:45:01.657771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.657745 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerStarted","Data":"6e6306739141f606e045aab866987b4ae9e218dc65529a8fc24a1339baa8fb8d"} Nov 01 05:45:01.657899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.657803 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerStarted","Data":"2def87d1c500fb0da73f4f39a7394d05fc5f6b6ae0f14a990e47145f93c3c0b5"} Nov 01 05:45:01.659870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.659837 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerStarted","Data":"ca489c951cebb43248378c72faf066ab9333d5ca491f36d9e9814416b16caa98"} Nov 01 05:45:01.659971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.659903 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerStarted","Data":"1b937e3f17e1be1ba1410a320496611977aeac8312c09125a759e80dd679c360"} Nov 01 05:45:01.661534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.661507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerStarted","Data":"2aa7f2d762f10398a5b24d1cbe458496704de97bd8f9f627474f76636e8bbd51"} Nov 01 05:45:01.663606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.663576 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerStarted","Data":"abf4a1ccc18af1788fd656a61906d4b37b6113a25de3903f5b474b4296ca0351"} Nov 01 05:45:01.773328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.773264 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" podStartSLOduration=1.773229759 podStartE2EDuration="1.773229759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:01.769026794 +0000 UTC m=+2516.909615713" watchObservedRunningTime="2026-11-01 05:45:01.773229759 +0000 UTC m=+2516.913818679" Nov 01 05:45:01.889360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:01.889300 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" podStartSLOduration=1.8892800269999999 podStartE2EDuration="1.889280027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:01.885180274 +0000 UTC m=+2517.025769216" watchObservedRunningTime="2026-11-01 05:45:01.889280027 +0000 UTC m=+2517.029868943" Nov 01 05:45:02.002047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.001988 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" podStartSLOduration=2.00197226 podStartE2EDuration="2.00197226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:02.000900528 +0000 UTC m=+2517.141489447" watchObservedRunningTime="2026-11-01 05:45:02.00197226 +0000 UTC m=+2517.142561220" Nov 01 05:45:02.672410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.672374 4425 generic.go:356] "Generic (PLEG): container finished" podID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerID="a939509a96d4918fc6c80a73569f9a047720f3d92892681fcf210059fb99eaae" exitCode=60 Nov 01 05:45:02.672906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.672453 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerDied","Data":"a939509a96d4918fc6c80a73569f9a047720f3d92892681fcf210059fb99eaae"} Nov 01 05:45:02.672906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.672794 4425 scope.go:122] "RemoveContainer" containerID="a939509a96d4918fc6c80a73569f9a047720f3d92892681fcf210059fb99eaae" Nov 01 05:45:02.674953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.674934 4425 generic.go:356] "Generic (PLEG): container finished" podID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerID="254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249" exitCode=0 Nov 01 05:45:02.675074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.675012 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerDied","Data":"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249"} Nov 01 05:45:02.675517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.675495 4425 scope.go:122] "RemoveContainer" containerID="254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249" Nov 01 05:45:02.676946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.676924 4425 generic.go:356] "Generic (PLEG): container finished" podID="9f219aa5-6c8b-458b-9918-f309fd517260" containerID="6e6306739141f606e045aab866987b4ae9e218dc65529a8fc24a1339baa8fb8d" exitCode=60 Nov 01 05:45:02.677058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.677011 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerDied","Data":"6e6306739141f606e045aab866987b4ae9e218dc65529a8fc24a1339baa8fb8d"} Nov 01 05:45:02.677319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.677247 4425 scope.go:122] "RemoveContainer" containerID="6e6306739141f606e045aab866987b4ae9e218dc65529a8fc24a1339baa8fb8d" Nov 01 05:45:02.677443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.677414 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:02.677524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.677506 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:02.677609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.677550 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:02.678814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.678781 4425 generic.go:356] "Generic (PLEG): container finished" podID="92474479-3d19-42de-a403-9a6136d4712f" containerID="ca489c951cebb43248378c72faf066ab9333d5ca491f36d9e9814416b16caa98" exitCode=60 Nov 01 05:45:02.678921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.678856 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerDied","Data":"ca489c951cebb43248378c72faf066ab9333d5ca491f36d9e9814416b16caa98"} Nov 01 05:45:02.679074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.679054 4425 scope.go:122] "RemoveContainer" containerID="ca489c951cebb43248378c72faf066ab9333d5ca491f36d9e9814416b16caa98" Nov 01 05:45:02.680707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.680688 4425 generic.go:356] "Generic (PLEG): container finished" podID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerID="ce8886009ac92320595c763588d79bf69cb0c5b0e2bdc6fdbffcb01c25a37d71" exitCode=60 Nov 01 05:45:02.680809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.680766 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerDied","Data":"ce8886009ac92320595c763588d79bf69cb0c5b0e2bdc6fdbffcb01c25a37d71"} Nov 01 05:45:02.680961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.680948 4425 scope.go:122] "RemoveContainer" containerID="ce8886009ac92320595c763588d79bf69cb0c5b0e2bdc6fdbffcb01c25a37d71" Nov 01 05:45:02.682456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.682434 4425 generic.go:356] "Generic (PLEG): container finished" podID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerID="3a9300c1d5c51df142cec1fad321f7a4977374559290479f8597662272cc91ab" exitCode=60 Nov 01 05:45:02.682574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.682521 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerDied","Data":"3a9300c1d5c51df142cec1fad321f7a4977374559290479f8597662272cc91ab"} Nov 01 05:45:02.682714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.682695 4425 scope.go:122] "RemoveContainer" containerID="3a9300c1d5c51df142cec1fad321f7a4977374559290479f8597662272cc91ab" Nov 01 05:45:02.686067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.685823 4425 generic.go:356] "Generic (PLEG): container finished" podID="3069f400-066e-4467-8855-6f97713c7470" containerID="f5c9ce282e91c7a63a78cb1e9e64adfbed2cf39e743d961f4b2e1fe96d7cbbd7" exitCode=60 Nov 01 05:45:02.686067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.685957 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerDied","Data":"f5c9ce282e91c7a63a78cb1e9e64adfbed2cf39e743d961f4b2e1fe96d7cbbd7"} Nov 01 05:45:02.686223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.686110 4425 scope.go:122] "RemoveContainer" containerID="f5c9ce282e91c7a63a78cb1e9e64adfbed2cf39e743d961f4b2e1fe96d7cbbd7" Nov 01 05:45:02.687888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.687854 4425 generic.go:356] "Generic (PLEG): container finished" podID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerID="8d7e57e8d37717218251cf1e3afe4ff2163ca9ccd51035caba3c6fdf24fcb043" exitCode=60 Nov 01 05:45:02.688236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.688137 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerDied","Data":"8d7e57e8d37717218251cf1e3afe4ff2163ca9ccd51035caba3c6fdf24fcb043"} Nov 01 05:45:02.688236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.688189 4425 scope.go:122] "RemoveContainer" containerID="8d7e57e8d37717218251cf1e3afe4ff2163ca9ccd51035caba3c6fdf24fcb043" Nov 01 05:45:02.689960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.689935 4425 generic.go:356] "Generic (PLEG): container finished" podID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerID="b79f937e967220b5aa16bab278b5c8f2e25dae0a78f99cb309f4fd021dbd0fc9" exitCode=60 Nov 01 05:45:02.690083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.689965 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerDied","Data":"b79f937e967220b5aa16bab278b5c8f2e25dae0a78f99cb309f4fd021dbd0fc9"} Nov 01 05:45:02.690372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.690236 4425 scope.go:122] "RemoveContainer" containerID="b79f937e967220b5aa16bab278b5c8f2e25dae0a78f99cb309f4fd021dbd0fc9" Nov 01 05:45:02.691861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.691837 4425 generic.go:356] "Generic (PLEG): container finished" podID="8356e552-9856-407e-b5ee-c4178e730b67" containerID="f19a6bdce111e4b663940377d002a262286108cdc722288895cbb9bf5d2f5fbd" exitCode=60 Nov 01 05:45:02.691963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.691876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerDied","Data":"f19a6bdce111e4b663940377d002a262286108cdc722288895cbb9bf5d2f5fbd"} Nov 01 05:45:02.692080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:02.692066 4425 scope.go:122] "RemoveContainer" containerID="f19a6bdce111e4b663940377d002a262286108cdc722288895cbb9bf5d2f5fbd" Nov 01 05:45:03.704886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.704789 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerStarted","Data":"b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a"} Nov 01 05:45:03.708106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.708076 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerStarted","Data":"90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe"} Nov 01 05:45:03.710997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.710966 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerStarted","Data":"d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998"} Nov 01 05:45:03.713288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.713208 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerStarted","Data":"a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15"} Nov 01 05:45:03.715541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.715512 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerStarted","Data":"946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40"} Nov 01 05:45:03.717423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.717398 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerStarted","Data":"00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8"} Nov 01 05:45:03.719842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.719820 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerStarted","Data":"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d"} Nov 01 05:45:03.721916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.721893 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerStarted","Data":"49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f"} Nov 01 05:45:03.723780 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.723760 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerStarted","Data":"81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a"} Nov 01 05:45:03.725864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:03.725844 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerStarted","Data":"6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0"} Nov 01 05:45:04.016636 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.016574 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" podStartSLOduration=4.016558419 podStartE2EDuration="4.016558419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.012905776 +0000 UTC m=+2519.153494693" watchObservedRunningTime="2026-11-01 05:45:04.016558419 +0000 UTC m=+2519.157147356" Nov 01 05:45:04.075961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.075923 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" podUID="4e7d9053-976a-4578-b5c5-470fa5966aea" containerName="payload-processing" containerID="cri-o://cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4" gracePeriod=25 Nov 01 05:45:04.084470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.084430 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerName="payload-processing" containerID="cri-o://5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63" gracePeriod=25 Nov 01 05:45:04.133150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.133099 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" podStartSLOduration=4.133086151 podStartE2EDuration="4.133086151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.132763593 +0000 UTC m=+2519.273352528" watchObservedRunningTime="2026-11-01 05:45:04.133086151 +0000 UTC m=+2519.273675088" Nov 01 05:45:04.378510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.378401 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" podStartSLOduration=4.378378709 podStartE2EDuration="4.378378709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.376082105 +0000 UTC m=+2519.516671026" watchObservedRunningTime="2026-11-01 05:45:04.378378709 +0000 UTC m=+2519.518967629" Nov 01 05:45:04.493427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.493366 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" podStartSLOduration=4.493347371 podStartE2EDuration="4.493347371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.492781928 +0000 UTC m=+2519.633370865" watchObservedRunningTime="2026-11-01 05:45:04.493347371 +0000 UTC m=+2519.633936292" Nov 01 05:45:04.498629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.498587 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:45:04.505607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.504780 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:45:04.525669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.525636 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:45:04.526305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.526243 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-rq68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 05:45:04.526447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.526334 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 05:45:04.621008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.620965 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\") pod \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " Nov 01 05:45:04.621197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\") pod \"4e7d9053-976a-4578-b5c5-470fa5966aea\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " Nov 01 05:45:04.621197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621111 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\") pod \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " Nov 01 05:45:04.621197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621153 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\") pod \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " Nov 01 05:45:04.621394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621342 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\") pod \"4e7d9053-976a-4578-b5c5-470fa5966aea\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " Nov 01 05:45:04.621394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621380 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\") pod \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\" (UID: \"964382ee-4646-4bc0-90fa-e13cf4d1d42a\") " Nov 01 05:45:04.621509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621446 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\") pod \"4e7d9053-976a-4578-b5c5-470fa5966aea\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " Nov 01 05:45:04.621509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621488 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\") pod \"4e7d9053-976a-4578-b5c5-470fa5966aea\" (UID: \"4e7d9053-976a-4578-b5c5-470fa5966aea\") " Nov 01 05:45:04.622216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621674 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume" pod "964382ee-4646-4bc0-90fa-e13cf4d1d42a" (UID: "964382ee-4646-4bc0-90fa-e13cf4d1d42a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:04.622216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621775 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume" pod "4e7d9053-976a-4578-b5c5-470fa5966aea" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:04.622216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.621944 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume" pod "4e7d9053-976a-4578-b5c5-470fa5966aea" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:04.622216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.622049 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume" pod "964382ee-4646-4bc0-90fa-e13cf4d1d42a" (UID: "964382ee-4646-4bc0-90fa-e13cf4d1d42a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:04.622483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.622237 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.622483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.622278 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.622483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.622293 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7d9053-976a-4578-b5c5-470fa5966aea-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.622483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.622306 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/964382ee-4646-4bc0-90fa-e13cf4d1d42a-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.623791 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.623766 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs" pod "964382ee-4646-4bc0-90fa-e13cf4d1d42a" (UID: "964382ee-4646-4bc0-90fa-e13cf4d1d42a"). InnerVolumeSpecName "kube-api-access-4vwcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:04.623898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.623802 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials" pod "964382ee-4646-4bc0-90fa-e13cf4d1d42a" (UID: "964382ee-4646-4bc0-90fa-e13cf4d1d42a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:04.624004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.623983 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r" pod "4e7d9053-976a-4578-b5c5-470fa5966aea" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea"). InnerVolumeSpecName "kube-api-access-x9c5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:04.624786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.624769 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials" pod "4e7d9053-976a-4578-b5c5-470fa5966aea" (UID: "4e7d9053-976a-4578-b5c5-470fa5966aea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:04.723788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.723748 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9c5r\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-kube-api-access-x9c5r\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.723788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.723788 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vwcs\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-kube-api-access-4vwcs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.724243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.723802 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e7d9053-976a-4578-b5c5-470fa5966aea-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.724243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.723816 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/964382ee-4646-4bc0-90fa-e13cf4d1d42a-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:04.735612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.735579 4425 generic.go:356] "Generic (PLEG): container finished" podID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerID="d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998" exitCode=60 Nov 01 05:45:04.735790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.735666 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerDied","Data":"d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998"} Nov 01 05:45:04.735790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.735743 4425 scope.go:122] "RemoveContainer" containerID="8d7e57e8d37717218251cf1e3afe4ff2163ca9ccd51035caba3c6fdf24fcb043" Nov 01 05:45:04.735976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.735958 4425 scope.go:122] "RemoveContainer" containerID="d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998" Nov 01 05:45:04.736351 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.736325 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8_odh-ai-gateway-infra(3fb12804-4ab5-4418-b081-cbd6a5708133)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" Nov 01 05:45:04.737873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.737841 4425 generic.go:356] "Generic (PLEG): container finished" podID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerID="a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15" exitCode=60 Nov 01 05:45:04.737982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.737914 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerDied","Data":"a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15"} Nov 01 05:45:04.738187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.738172 4425 scope.go:122] "RemoveContainer" containerID="a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15" Nov 01 05:45:04.738570 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.738494 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg_odh-ai-gateway-infra(99a0be2c-7ffc-4892-9b04-c7454b8a780e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" Nov 01 05:45:04.741173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.741149 4425 generic.go:356] "Generic (PLEG): container finished" podID="8356e552-9856-407e-b5ee-c4178e730b67" containerID="946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40" exitCode=60 Nov 01 05:45:04.741301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.741228 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerDied","Data":"946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40"} Nov 01 05:45:04.764336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.758666 4425 scope.go:122] "RemoveContainer" containerID="946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40" Nov 01 05:45:04.767068 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.767029 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5_odh-ai-gateway-infra(8356e552-9856-407e-b5ee-c4178e730b67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" podUID="8356e552-9856-407e-b5ee-c4178e730b67" Nov 01 05:45:04.769188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.769108 4425 generic.go:356] "Generic (PLEG): container finished" podID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerID="00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8" exitCode=60 Nov 01 05:45:04.769872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.769699 4425 scope.go:122] "RemoveContainer" containerID="00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8" Nov 01 05:45:04.770770 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.770063 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh_odh-ai-gateway-infra(5dcc7037-d1d1-4af6-9295-3e91e15c7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" Nov 01 05:45:04.770770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.770121 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerDied","Data":"00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8"} Nov 01 05:45:04.773900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.773876 4425 generic.go:356] "Generic (PLEG): container finished" podID="9f219aa5-6c8b-458b-9918-f309fd517260" containerID="49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f" exitCode=60 Nov 01 05:45:04.774006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.773981 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerDied","Data":"49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f"} Nov 01 05:45:04.774228 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.774212 4425 scope.go:122] "RemoveContainer" containerID="49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f" Nov 01 05:45:04.774529 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.774508 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz_odh-ai-gateway-infra(9f219aa5-6c8b-458b-9918-f309fd517260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" Nov 01 05:45:04.776180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.776155 4425 generic.go:356] "Generic (PLEG): container finished" podID="92474479-3d19-42de-a403-9a6136d4712f" containerID="81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a" exitCode=60 Nov 01 05:45:04.776317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.776238 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerDied","Data":"81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a"} Nov 01 05:45:04.776533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.776512 4425 scope.go:122] "RemoveContainer" containerID="81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a" Nov 01 05:45:04.776856 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.776831 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b_odh-ai-gateway-infra(92474479-3d19-42de-a403-9a6136d4712f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" podUID="92474479-3d19-42de-a403-9a6136d4712f" Nov 01 05:45:04.777820 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.777794 4425 generic.go:356] "Generic (PLEG): container finished" podID="4e7d9053-976a-4578-b5c5-470fa5966aea" containerID="cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4" exitCode=0 Nov 01 05:45:04.777923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.777822 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" event={"ID":"4e7d9053-976a-4578-b5c5-470fa5966aea","Type":"ContainerDied","Data":"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4"} Nov 01 05:45:04.777923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.777851 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" Nov 01 05:45:04.777923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.777861 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf" event={"ID":"4e7d9053-976a-4578-b5c5-470fa5966aea","Type":"ContainerDied","Data":"a2dcb6391ba62ca20ff6958d5414e94370c272f0ce84e95902ecaf6418422757"} Nov 01 05:45:04.780102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.780076 4425 generic.go:356] "Generic (PLEG): container finished" podID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerID="6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0" exitCode=60 Nov 01 05:45:04.780372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.780136 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerDied","Data":"6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0"} Nov 01 05:45:04.780480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.780459 4425 scope.go:122] "RemoveContainer" containerID="6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0" Nov 01 05:45:04.780785 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.780763 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt_odh-ai-gateway-infra(0a0d7b47-92c1-4cc4-b539-c54166a0ebdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" Nov 01 05:45:04.781583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.781562 4425 generic.go:356] "Generic (PLEG): container finished" podID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerID="5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63" exitCode=0 Nov 01 05:45:04.781677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.781607 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" Nov 01 05:45:04.781677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.781647 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" event={"ID":"964382ee-4646-4bc0-90fa-e13cf4d1d42a","Type":"ContainerDied","Data":"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63"} Nov 01 05:45:04.781802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.781708 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7" event={"ID":"964382ee-4646-4bc0-90fa-e13cf4d1d42a","Type":"ContainerDied","Data":"eea27c9870bae52edc3b2968bc16aa836bb518297f4d40d38e628f61d8e39873"} Nov 01 05:45:04.783917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.783892 4425 generic.go:356] "Generic (PLEG): container finished" podID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerID="b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a" exitCode=60 Nov 01 05:45:04.784027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.783969 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerDied","Data":"b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a"} Nov 01 05:45:04.784204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.784187 4425 scope.go:122] "RemoveContainer" containerID="b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a" Nov 01 05:45:04.784461 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.784442 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891865-xgs2q_odh-ai-gateway-infra(2a137f1f-902a-44ed-9840-63fbfc34aa25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" Nov 01 05:45:04.788669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.788563 4425 generic.go:356] "Generic (PLEG): container finished" podID="3069f400-066e-4467-8855-6f97713c7470" containerID="90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe" exitCode=60 Nov 01 05:45:04.788669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.788652 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerDied","Data":"90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe"} Nov 01 05:45:04.788918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.788901 4425 scope.go:122] "RemoveContainer" containerID="90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe" Nov 01 05:45:04.789178 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.789157 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q_odh-ai-gateway-infra(3069f400-066e-4467-8855-6f97713c7470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" podUID="3069f400-066e-4467-8855-6f97713c7470" Nov 01 05:45:04.789384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.789370 4425 scope.go:122] "RemoveContainer" containerID="b79f937e967220b5aa16bab278b5c8f2e25dae0a78f99cb309f4fd021dbd0fc9" Nov 01 05:45:04.803655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.803636 4425 scope.go:122] "RemoveContainer" containerID="f19a6bdce111e4b663940377d002a262286108cdc722288895cbb9bf5d2f5fbd" Nov 01 05:45:04.817126 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.817106 4425 scope.go:122] "RemoveContainer" containerID="a939509a96d4918fc6c80a73569f9a047720f3d92892681fcf210059fb99eaae" Nov 01 05:45:04.831442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.831422 4425 scope.go:122] "RemoveContainer" containerID="6e6306739141f606e045aab866987b4ae9e218dc65529a8fc24a1339baa8fb8d" Nov 01 05:45:04.867306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.856646 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" podStartSLOduration=4.856626913 podStartE2EDuration="4.856626913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:04.856155335 +0000 UTC m=+2519.996744267" watchObservedRunningTime="2026-11-01 05:45:04.856626913 +0000 UTC m=+2519.997215834" Nov 01 05:45:04.870762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.870741 4425 scope.go:122] "RemoveContainer" containerID="ca489c951cebb43248378c72faf066ab9333d5ca491f36d9e9814416b16caa98" Nov 01 05:45:04.885762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.885732 4425 scope.go:122] "RemoveContainer" containerID="cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4" Nov 01 05:45:04.895430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.895410 4425 scope.go:122] "RemoveContainer" containerID="cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4" Nov 01 05:45:04.895747 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.895719 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4\": container with ID starting with cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4 not found: ID does not exist" containerID="cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4" Nov 01 05:45:04.895796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.895757 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4"} err="failed to get container status \"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4\": rpc error: code = NotFound desc = could not find container \"cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4\": container with ID starting with cb2eab261b2b74d4a6473160b294bcc7895809a2acdfb39aef2a20f1352b10f4 not found: ID does not exist" Nov 01 05:45:04.895796 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.895778 4425 scope.go:122] "RemoveContainer" containerID="ce8886009ac92320595c763588d79bf69cb0c5b0e2bdc6fdbffcb01c25a37d71" Nov 01 05:45:04.909971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.909943 4425 scope.go:122] "RemoveContainer" containerID="5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63" Nov 01 05:45:04.921429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.921406 4425 scope.go:122] "RemoveContainer" containerID="5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63" Nov 01 05:45:04.921743 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:04.921721 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63\": container with ID starting with 5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63 not found: ID does not exist" containerID="5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63" Nov 01 05:45:04.921842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.921759 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63"} err="failed to get container status \"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63\": rpc error: code = NotFound desc = could not find container \"5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63\": container with ID starting with 5c53772e2169df9dd8dfa8e82f08d6ebfc767964b1a1519d056601ec69509a63 not found: ID does not exist" Nov 01 05:45:04.921842 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.921784 4425 scope.go:122] "RemoveContainer" containerID="3a9300c1d5c51df142cec1fad321f7a4977374559290479f8597662272cc91ab" Nov 01 05:45:04.936210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:04.936184 4425 scope.go:122] "RemoveContainer" containerID="f5c9ce282e91c7a63a78cb1e9e64adfbed2cf39e743d961f4b2e1fe96d7cbbd7" Nov 01 05:45:05.093209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.093143 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" podStartSLOduration=5.093129456 podStartE2EDuration="5.093129456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:05.089472716 +0000 UTC m=+2520.230061646" watchObservedRunningTime="2026-11-01 05:45:05.093129456 +0000 UTC m=+2520.233718372" Nov 01 05:45:05.526770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.526685 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-rq68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 05:45:05.526770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.526745 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 05:45:05.799826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.799742 4425 scope.go:122] "RemoveContainer" containerID="00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8" Nov 01 05:45:05.800293 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.800049 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh_odh-ai-gateway-infra(5dcc7037-d1d1-4af6-9295-3e91e15c7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" Nov 01 05:45:05.801582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.801562 4425 scope.go:122] "RemoveContainer" containerID="49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f" Nov 01 05:45:05.801893 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.801868 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz_odh-ai-gateway-infra(9f219aa5-6c8b-458b-9918-f309fd517260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" Nov 01 05:45:05.803367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.803348 4425 scope.go:122] "RemoveContainer" containerID="81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a" Nov 01 05:45:05.803639 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.803615 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b_odh-ai-gateway-infra(92474479-3d19-42de-a403-9a6136d4712f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" podUID="92474479-3d19-42de-a403-9a6136d4712f" Nov 01 05:45:05.805911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.805894 4425 scope.go:122] "RemoveContainer" containerID="6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0" Nov 01 05:45:05.806143 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.806122 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt_odh-ai-gateway-infra(0a0d7b47-92c1-4cc4-b539-c54166a0ebdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" Nov 01 05:45:05.808384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.808365 4425 scope.go:122] "RemoveContainer" containerID="b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a" Nov 01 05:45:05.808624 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.808593 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891865-xgs2q_odh-ai-gateway-infra(2a137f1f-902a-44ed-9840-63fbfc34aa25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" Nov 01 05:45:05.810013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.809994 4425 scope.go:122] "RemoveContainer" containerID="90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe" Nov 01 05:45:05.810262 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.810228 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q_odh-ai-gateway-infra(3069f400-066e-4467-8855-6f97713c7470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" podUID="3069f400-066e-4467-8855-6f97713c7470" Nov 01 05:45:05.811492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.811474 4425 scope.go:122] "RemoveContainer" containerID="d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998" Nov 01 05:45:05.811711 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.811686 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8_odh-ai-gateway-infra(3fb12804-4ab5-4418-b081-cbd6a5708133)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" Nov 01 05:45:05.812977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.812959 4425 scope.go:122] "RemoveContainer" containerID="a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15" Nov 01 05:45:05.813192 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.813170 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg_odh-ai-gateway-infra(99a0be2c-7ffc-4892-9b04-c7454b8a780e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" Nov 01 05:45:05.814472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:05.814458 4425 scope.go:122] "RemoveContainer" containerID="946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40" Nov 01 05:45:05.814633 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:05.814617 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5_odh-ai-gateway-infra(8356e552-9856-407e-b5ee-c4178e730b67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" podUID="8356e552-9856-407e-b5ee-c4178e730b67" Nov 01 05:45:06.040574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.040534 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:45:06.071400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.071364 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-686b7njmf"] Nov 01 05:45:06.370658 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.370552 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:45:06.374737 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.374710 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4d5cph7"] Nov 01 05:45:06.525957 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.525919 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-rq68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 05:45:06.526131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.525974 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 05:45:06.975783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.975747 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7d9053-976a-4578-b5c5-470fa5966aea" path="/var/lib/kubelet/pods/4e7d9053-976a-4578-b5c5-470fa5966aea/volumes" Nov 01 05:45:06.976599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:06.976310 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" path="/var/lib/kubelet/pods/964382ee-4646-4bc0-90fa-e13cf4d1d42a/volumes" Nov 01 05:45:07.526092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:07.526053 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-rq68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 05:45:07.526310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:07.526122 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 05:45:08.526709 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:08.526668 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-rq68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 05:45:08.527135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:08.526741 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 05:45:08.970089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:08.970038 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:09.531215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:09.531183 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:45:09.531634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:09.531617 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:45:09.532315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:09.532295 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:45:14.979059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:14.978744 4425 scope.go:122] "RemoveContainer" containerID="90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe" Nov 01 05:45:14.979059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:14.978952 4425 scope.go:122] "RemoveContainer" containerID="49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f" Nov 01 05:45:15.896395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.896362 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerStarted","Data":"14952ebaf3d1c47af1b9d5490f9ed4828bf8062b346c972cc460192b3e5f7447"} Nov 01 05:45:15.898501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.898472 4425 generic.go:356] "Generic (PLEG): container finished" podID="9f219aa5-6c8b-458b-9918-f309fd517260" containerID="1d298fd7695965563edfd39ec612e20d1712844ab8f1b262514c79f3ffc3e98b" exitCode=60 Nov 01 05:45:15.898624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.898538 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerDied","Data":"1d298fd7695965563edfd39ec612e20d1712844ab8f1b262514c79f3ffc3e98b"} Nov 01 05:45:15.898624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.898588 4425 scope.go:122] "RemoveContainer" containerID="49a571734504a97b2bb3c64c3863ce510b4ac5d3670ee57f348fcdb2e30e276f" Nov 01 05:45:15.898949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.898928 4425 scope.go:122] "RemoveContainer" containerID="1d298fd7695965563edfd39ec612e20d1712844ab8f1b262514c79f3ffc3e98b" Nov 01 05:45:15.899234 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:15.899208 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz_odh-ai-gateway-infra(9f219aa5-6c8b-458b-9918-f309fd517260)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" Nov 01 05:45:15.966661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.966638 4425 scope.go:122] "RemoveContainer" containerID="a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15" Nov 01 05:45:15.966789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:15.966734 4425 scope.go:122] "RemoveContainer" containerID="b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a" Nov 01 05:45:16.909449 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.909409 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerStarted","Data":"f497f1d8f8a876e3a0add5fe074d931b242c864e1f80f3798edaa90cdfb8690b"} Nov 01 05:45:16.911632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.911604 4425 generic.go:356] "Generic (PLEG): container finished" podID="3069f400-066e-4467-8855-6f97713c7470" containerID="14952ebaf3d1c47af1b9d5490f9ed4828bf8062b346c972cc460192b3e5f7447" exitCode=60 Nov 01 05:45:16.911798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.911683 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerDied","Data":"14952ebaf3d1c47af1b9d5490f9ed4828bf8062b346c972cc460192b3e5f7447"} Nov 01 05:45:16.911798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.911732 4425 scope.go:122] "RemoveContainer" containerID="90e0c39d23422f3f992fe4e53720892a2fcea8b1cbab8ab90ab7362b6f448cfe" Nov 01 05:45:16.912086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.912068 4425 scope.go:122] "RemoveContainer" containerID="14952ebaf3d1c47af1b9d5490f9ed4828bf8062b346c972cc460192b3e5f7447" Nov 01 05:45:16.912393 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:16.912372 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q_odh-ai-gateway-infra(3069f400-066e-4467-8855-6f97713c7470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" podUID="3069f400-066e-4467-8855-6f97713c7470" Nov 01 05:45:16.913817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:16.913797 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerStarted","Data":"72df7c97570da5d5b461718c99a9814470b391bd747de98e5aee11ca05ea34c6"} Nov 01 05:45:17.025382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.025348 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:17.142732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.142687 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:17.318683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.318648 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:17.412191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.412092 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\") pod \"9f219aa5-6c8b-458b-9918-f309fd517260\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " Nov 01 05:45:17.412422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.412357 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\" (UniqueName: \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\") pod \"9f219aa5-6c8b-458b-9918-f309fd517260\" (UID: \"9f219aa5-6c8b-458b-9918-f309fd517260\") " Nov 01 05:45:17.412621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.412594 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca" pod "9f219aa5-6c8b-458b-9918-f309fd517260" (UID: "9f219aa5-6c8b-458b-9918-f309fd517260"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:17.412896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.412877 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f219aa5-6c8b-458b-9918-f309fd517260-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:17.414706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.414686 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj" pod "9f219aa5-6c8b-458b-9918-f309fd517260" (UID: "9f219aa5-6c8b-458b-9918-f309fd517260"). InnerVolumeSpecName "kube-api-access-7rpvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:17.515881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.515849 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rpvj\" (UniqueName: \"kubernetes.io/projected/9f219aa5-6c8b-458b-9918-f309fd517260-kube-api-access-7rpvj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:17.923206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.923173 4425 generic.go:356] "Generic (PLEG): container finished" podID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerID="f497f1d8f8a876e3a0add5fe074d931b242c864e1f80f3798edaa90cdfb8690b" exitCode=60 Nov 01 05:45:17.923750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.923246 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerDied","Data":"f497f1d8f8a876e3a0add5fe074d931b242c864e1f80f3798edaa90cdfb8690b"} Nov 01 05:45:17.923750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.923348 4425 scope.go:122] "RemoveContainer" containerID="b038277666446c36aa079bfbd096d98b26f5d48f747482107efba1e52632fa9a" Nov 01 05:45:17.923750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.923705 4425 scope.go:122] "RemoveContainer" containerID="f497f1d8f8a876e3a0add5fe074d931b242c864e1f80f3798edaa90cdfb8690b" Nov 01 05:45:17.924003 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:17.923983 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891865-xgs2q_odh-ai-gateway-infra(2a137f1f-902a-44ed-9840-63fbfc34aa25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" Nov 01 05:45:17.927288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.927237 4425 generic.go:356] "Generic (PLEG): container finished" podID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerID="72df7c97570da5d5b461718c99a9814470b391bd747de98e5aee11ca05ea34c6" exitCode=60 Nov 01 05:45:17.927417 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.927295 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerDied","Data":"72df7c97570da5d5b461718c99a9814470b391bd747de98e5aee11ca05ea34c6"} Nov 01 05:45:17.927711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.927688 4425 scope.go:122] "RemoveContainer" containerID="72df7c97570da5d5b461718c99a9814470b391bd747de98e5aee11ca05ea34c6" Nov 01 05:45:17.928024 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:17.928004 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg_odh-ai-gateway-infra(99a0be2c-7ffc-4892-9b04-c7454b8a780e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" Nov 01 05:45:17.929091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.929059 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" event={"ID":"9f219aa5-6c8b-458b-9918-f309fd517260","Type":"ContainerDied","Data":"2def87d1c500fb0da73f4f39a7394d05fc5f6b6ae0f14a990e47145f93c3c0b5"} Nov 01 05:45:17.929091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.929082 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz" Nov 01 05:45:17.940616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.940596 4425 scope.go:122] "RemoveContainer" containerID="a882283260592d89345e20086a2f24579ceb1b1dae4a6d087ef66a378e798b15" Nov 01 05:45:17.954546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.954521 4425 scope.go:122] "RemoveContainer" containerID="1d298fd7695965563edfd39ec612e20d1712844ab8f1b262514c79f3ffc3e98b" Nov 01 05:45:17.966502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.966479 4425 scope.go:122] "RemoveContainer" containerID="00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8" Nov 01 05:45:17.966646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.966629 4425 scope.go:122] "RemoveContainer" containerID="6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0" Nov 01 05:45:17.966730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.966708 4425 scope.go:122] "RemoveContainer" containerID="946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40" Nov 01 05:45:17.966792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.966750 4425 scope.go:122] "RemoveContainer" containerID="d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998" Nov 01 05:45:17.966947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:17.966923 4425 scope.go:122] "RemoveContainer" containerID="81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a" Nov 01 05:45:18.041688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.041657 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:18.277881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.277844 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:18.290750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.290709 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:18.298910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.298887 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891865-qzrlz"] Nov 01 05:45:18.307105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.307083 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:18.337124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337090 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:45:18.337761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337746 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.337807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337762 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.337807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337784 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4e7d9053-976a-4578-b5c5-470fa5966aea" containerName="payload-processing" Nov 01 05:45:18.337807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337790 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7d9053-976a-4578-b5c5-470fa5966aea" containerName="payload-processing" Nov 01 05:45:18.337807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337800 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337805 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337826 4425 cpu_manager.go:414] "containerMap: removing container" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerName="payload-processing" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337830 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerName="payload-processing" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337841 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337845 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337851 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337856 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337872 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337876 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337956 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337966 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7d9053-976a-4578-b5c5-470fa5966aea" containerName="payload-processing" Nov 01 05:45:18.337978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337974 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.338302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337983 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="964382ee-4646-4bc0-90fa-e13cf4d1d42a" containerName="payload-processing" Nov 01 05:45:18.338302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337990 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:18.338302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.337997 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:18.341529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.341509 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.395172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395105 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 05:45:18.395172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395105 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 05:45:18.395172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395164 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 05:45:18.395522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395191 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3de4e179dc63\"" Nov 01 05:45:18.395522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395233 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-j9xvt\"" Nov 01 05:45:18.395522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.395240 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 05:45:18.399312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.399287 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:45:18.443964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.443929 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\" (UniqueName: \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\") pod \"3069f400-066e-4467-8855-6f97713c7470\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " Nov 01 05:45:18.444168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444146 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\") pod \"3069f400-066e-4467-8855-6f97713c7470\" (UID: \"3069f400-066e-4467-8855-6f97713c7470\") " Nov 01 05:45:18.444457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444433 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.444558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444510 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca" pod "3069f400-066e-4467-8855-6f97713c7470" (UID: "3069f400-066e-4467-8855-6f97713c7470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:18.444631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444582 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.444689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444637 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.444797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444777 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnn7f\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.444961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.444942 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069f400-066e-4467-8855-6f97713c7470-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:18.490508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.490475 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:45:18.542242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.542007 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc" pod "3069f400-066e-4467-8855-6f97713c7470" (UID: "3069f400-066e-4467-8855-6f97713c7470"). InnerVolumeSpecName "kube-api-access-p6kxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:18.546072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.546023 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnn7f\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.546230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.546197 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.546320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.546285 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.546376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.546354 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.546939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.546465 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kxc\" (UniqueName: \"kubernetes.io/projected/3069f400-066e-4467-8855-6f97713c7470-kube-api-access-p6kxc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:18.547529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.547500 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.548424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.548393 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.549867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.549839 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.599545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.599511 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:45:18.610430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.610401 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnn7f\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\") pod \"payload-processing-external-model-67d67f47f4-bp4gq\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.668023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.667991 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:18.948641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.948604 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" event={"ID":"3069f400-066e-4467-8855-6f97713c7470","Type":"ContainerDied","Data":"cc60f64ef45acff6183612e3d8a01878d64b49e10bc0c4af8373a3af754ce2ee"} Nov 01 05:45:18.948967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.948683 4425 scope.go:122] "RemoveContainer" containerID="14952ebaf3d1c47af1b9d5490f9ed4828bf8062b346c972cc460192b3e5f7447" Nov 01 05:45:18.948967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.948791 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q" Nov 01 05:45:18.959272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.959230 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerStarted","Data":"31b3e0fd8489437cbd195a315d4c9bffdc5d132c38b556129c35924ce4601c82"} Nov 01 05:45:18.974509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.974452 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:18.998843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.998804 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" path="/var/lib/kubelet/pods/9f219aa5-6c8b-458b-9918-f309fd517260/volumes" Nov 01 05:45:18.999285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.999221 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerStarted","Data":"59c88add8afeb8edc180d73a9f44f8fe961132952c7128d234f88c0e9dcb286e"} Nov 01 05:45:18.999389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.999302 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerStarted","Data":"5a674834e2886fad27b0a607c1cdf20d411938c1b279b51951cccefaee65ff13"} Nov 01 05:45:18.999389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:18.999334 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerStarted","Data":"ba9bbe11971b1f8f82dfad2b80006d4b8c708d41c69da3f1e500d6375130f8fc"} Nov 01 05:45:19.001048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.001022 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerStarted","Data":"318446af23ffc52f6cc3e34013c609c64cf998860d28dfa78a787da8b7a76fae"} Nov 01 05:45:19.074535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.074505 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:45:19.238659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.238633 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:19.372074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.372052 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\" (UniqueName: \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\") pod \"2a137f1f-902a-44ed-9840-63fbfc34aa25\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " Nov 01 05:45:19.372217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.372123 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\") pod \"2a137f1f-902a-44ed-9840-63fbfc34aa25\" (UID: \"2a137f1f-902a-44ed-9840-63fbfc34aa25\") " Nov 01 05:45:19.372533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.372497 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca" pod "2a137f1f-902a-44ed-9840-63fbfc34aa25" (UID: "2a137f1f-902a-44ed-9840-63fbfc34aa25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:19.374358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.374335 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx" pod "2a137f1f-902a-44ed-9840-63fbfc34aa25" (UID: "2a137f1f-902a-44ed-9840-63fbfc34aa25"). InnerVolumeSpecName "kube-api-access-vvlxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:19.425052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.425018 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:19.435664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.435634 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891865-b7g9q"] Nov 01 05:45:19.436194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.436173 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:19.474213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.474145 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlxx\" (UniqueName: \"kubernetes.io/projected/2a137f1f-902a-44ed-9840-63fbfc34aa25-kube-api-access-vvlxx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:19.474213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.474169 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a137f1f-902a-44ed-9840-63fbfc34aa25-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:19.574806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.574770 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\") pod \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " Nov 01 05:45:19.574991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.574815 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\" (UniqueName: \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\") pod \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\" (UID: \"99a0be2c-7ffc-4892-9b04-c7454b8a780e\") " Nov 01 05:45:19.575241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.575208 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca" pod "99a0be2c-7ffc-4892-9b04-c7454b8a780e" (UID: "99a0be2c-7ffc-4892-9b04-c7454b8a780e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:19.577089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.577062 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw" pod "99a0be2c-7ffc-4892-9b04-c7454b8a780e" (UID: "99a0be2c-7ffc-4892-9b04-c7454b8a780e"). InnerVolumeSpecName "kube-api-access-x9fjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:19.676888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.676851 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a0be2c-7ffc-4892-9b04-c7454b8a780e-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:19.676888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:19.676880 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fjw\" (UniqueName: \"kubernetes.io/projected/99a0be2c-7ffc-4892-9b04-c7454b8a780e-kube-api-access-x9fjw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:20.010950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.010917 4425 generic.go:356] "Generic (PLEG): container finished" podID="92474479-3d19-42de-a403-9a6136d4712f" containerID="ba9bbe11971b1f8f82dfad2b80006d4b8c708d41c69da3f1e500d6375130f8fc" exitCode=60 Nov 01 05:45:20.011438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.010984 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerDied","Data":"ba9bbe11971b1f8f82dfad2b80006d4b8c708d41c69da3f1e500d6375130f8fc"} Nov 01 05:45:20.011438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.011045 4425 scope.go:122] "RemoveContainer" containerID="81113b63cb4bf1a79e21e21e485e08c0e1ee074932a9fe5e887e138b6af8e95a" Nov 01 05:45:20.011583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.011456 4425 scope.go:122] "RemoveContainer" containerID="ba9bbe11971b1f8f82dfad2b80006d4b8c708d41c69da3f1e500d6375130f8fc" Nov 01 05:45:20.011799 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:20.011769 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b_odh-ai-gateway-infra(92474479-3d19-42de-a403-9a6136d4712f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" podUID="92474479-3d19-42de-a403-9a6136d4712f" Nov 01 05:45:20.013369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.013350 4425 generic.go:356] "Generic (PLEG): container finished" podID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerID="318446af23ffc52f6cc3e34013c609c64cf998860d28dfa78a787da8b7a76fae" exitCode=60 Nov 01 05:45:20.013463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.013419 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerDied","Data":"318446af23ffc52f6cc3e34013c609c64cf998860d28dfa78a787da8b7a76fae"} Nov 01 05:45:20.013735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.013720 4425 scope.go:122] "RemoveContainer" containerID="318446af23ffc52f6cc3e34013c609c64cf998860d28dfa78a787da8b7a76fae" Nov 01 05:45:20.014009 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:20.013990 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt_odh-ai-gateway-infra(0a0d7b47-92c1-4cc4-b539-c54166a0ebdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" Nov 01 05:45:20.015035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.015013 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" event={"ID":"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d","Type":"ContainerStarted","Data":"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5"} Nov 01 05:45:20.015173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.015069 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" event={"ID":"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d","Type":"ContainerStarted","Data":"0495a344a4f526917b09fc5850a04b74033d899c1776346d44f556b4ab5ff6ec"} Nov 01 05:45:20.015231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.015153 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:20.016945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.016926 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" event={"ID":"2a137f1f-902a-44ed-9840-63fbfc34aa25","Type":"ContainerDied","Data":"abf4a1ccc18af1788fd656a61906d4b37b6113a25de3903f5b474b4296ca0351"} Nov 01 05:45:20.017063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.016931 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q" Nov 01 05:45:20.022906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.022877 4425 generic.go:356] "Generic (PLEG): container finished" podID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerID="31b3e0fd8489437cbd195a315d4c9bffdc5d132c38b556129c35924ce4601c82" exitCode=60 Nov 01 05:45:20.023030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.022956 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerDied","Data":"31b3e0fd8489437cbd195a315d4c9bffdc5d132c38b556129c35924ce4601c82"} Nov 01 05:45:20.023330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.023314 4425 scope.go:122] "RemoveContainer" containerID="31b3e0fd8489437cbd195a315d4c9bffdc5d132c38b556129c35924ce4601c82" Nov 01 05:45:20.023588 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:20.023570 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8_odh-ai-gateway-infra(3fb12804-4ab5-4418-b081-cbd6a5708133)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" Nov 01 05:45:20.024849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.024822 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" Nov 01 05:45:20.024954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.024842 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg" event={"ID":"99a0be2c-7ffc-4892-9b04-c7454b8a780e","Type":"ContainerDied","Data":"78e81a5060b080ccdec70b9b695780035882bec61aba12852a6415da1dc43741"} Nov 01 05:45:20.027164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.027140 4425 generic.go:356] "Generic (PLEG): container finished" podID="8356e552-9856-407e-b5ee-c4178e730b67" containerID="59c88add8afeb8edc180d73a9f44f8fe961132952c7128d234f88c0e9dcb286e" exitCode=60 Nov 01 05:45:20.027323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.027217 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerDied","Data":"59c88add8afeb8edc180d73a9f44f8fe961132952c7128d234f88c0e9dcb286e"} Nov 01 05:45:20.027482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.027412 4425 scope.go:122] "RemoveContainer" containerID="6a31befdfefac3c796d65c228cc16913beff99eef1e11777189251e55f9969e0" Nov 01 05:45:20.027648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.027630 4425 scope.go:122] "RemoveContainer" containerID="59c88add8afeb8edc180d73a9f44f8fe961132952c7128d234f88c0e9dcb286e" Nov 01 05:45:20.027927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:20.027901 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5_odh-ai-gateway-infra(8356e552-9856-407e-b5ee-c4178e730b67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" podUID="8356e552-9856-407e-b5ee-c4178e730b67" Nov 01 05:45:20.029509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.029483 4425 generic.go:356] "Generic (PLEG): container finished" podID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerID="5a674834e2886fad27b0a607c1cdf20d411938c1b279b51951cccefaee65ff13" exitCode=60 Nov 01 05:45:20.029601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.029522 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerDied","Data":"5a674834e2886fad27b0a607c1cdf20d411938c1b279b51951cccefaee65ff13"} Nov 01 05:45:20.029866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.029850 4425 scope.go:122] "RemoveContainer" containerID="5a674834e2886fad27b0a607c1cdf20d411938c1b279b51951cccefaee65ff13" Nov 01 05:45:20.030107 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:20.030089 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh_odh-ai-gateway-infra(5dcc7037-d1d1-4af6-9295-3e91e15c7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" Nov 01 05:45:20.046947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.046918 4425 scope.go:122] "RemoveContainer" containerID="f497f1d8f8a876e3a0add5fe074d931b242c864e1f80f3798edaa90cdfb8690b" Nov 01 05:45:20.071066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.071037 4425 scope.go:122] "RemoveContainer" containerID="d805f1108a6e17aa756e889fa5660ef4ef493a03c4ceb6a7817577c7ed9b6998" Nov 01 05:45:20.085241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.085216 4425 scope.go:122] "RemoveContainer" containerID="72df7c97570da5d5b461718c99a9814470b391bd747de98e5aee11ca05ea34c6" Nov 01 05:45:20.099020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.098995 4425 scope.go:122] "RemoveContainer" containerID="946cf68c41d8818845d019275e712a19edb6ebe4cd8f6ac714b69eb1e8722e40" Nov 01 05:45:20.114259 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.114229 4425 scope.go:122] "RemoveContainer" containerID="00d66087448e750662de723b1361979e759f73fb1cc63197728a1385fa2b58e8" Nov 01 05:45:20.199626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.199595 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:20.321032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.321003 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:20.378846 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.378788 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" podStartSLOduration=2.378772069 podStartE2EDuration="2.378772069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:20.377454878 +0000 UTC m=+2535.518043819" watchObservedRunningTime="2026-11-01 05:45:20.378772069 +0000 UTC m=+2535.519360987" Nov 01 05:45:20.608470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.608383 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:20.612465 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.612435 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891865-cshbg"] Nov 01 05:45:20.674538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.674510 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:20.792720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.792686 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:20.908247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.908163 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:20.975348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.975316 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3069f400-066e-4467-8855-6f97713c7470" path="/var/lib/kubelet/pods/3069f400-066e-4467-8855-6f97713c7470/volumes" Nov 01 05:45:20.975776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:20.975762 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" path="/var/lib/kubelet/pods/99a0be2c-7ffc-4892-9b04-c7454b8a780e/volumes" Nov 01 05:45:21.085285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.075603 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:21.085285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.083435 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891865-xgs2q"] Nov 01 05:45:21.241571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.241438 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:21.284872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.284841 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:21.287690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.287647 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:21.327657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.327626 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:21.335164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.335142 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:21.400870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.400838 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\") pod \"3fb12804-4ab5-4418-b081-cbd6a5708133\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " Nov 01 05:45:21.401038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.400960 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\") pod \"8356e552-9856-407e-b5ee-c4178e730b67\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " Nov 01 05:45:21.401038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401000 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\" (UniqueName: \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\") pod \"3fb12804-4ab5-4418-b081-cbd6a5708133\" (UID: \"3fb12804-4ab5-4418-b081-cbd6a5708133\") " Nov 01 05:45:21.401038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401023 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\") pod \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " Nov 01 05:45:21.401190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401102 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\" (UniqueName: \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\") pod \"8356e552-9856-407e-b5ee-c4178e730b67\" (UID: \"8356e552-9856-407e-b5ee-c4178e730b67\") " Nov 01 05:45:21.401238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401149 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\" (UniqueName: \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\") pod \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\" (UID: \"5dcc7037-d1d1-4af6-9295-3e91e15c7043\") " Nov 01 05:45:21.401431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401405 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca" pod "3fb12804-4ab5-4418-b081-cbd6a5708133" (UID: "3fb12804-4ab5-4418-b081-cbd6a5708133"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:21.401497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401434 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca" pod "8356e552-9856-407e-b5ee-c4178e730b67" (UID: "8356e552-9856-407e-b5ee-c4178e730b67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:21.401497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401469 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca" pod "5dcc7037-d1d1-4af6-9295-3e91e15c7043" (UID: "5dcc7037-d1d1-4af6-9295-3e91e15c7043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:21.401807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401755 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb12804-4ab5-4418-b081-cbd6a5708133-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.401807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401774 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356e552-9856-407e-b5ee-c4178e730b67-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.401807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.401784 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dcc7037-d1d1-4af6-9295-3e91e15c7043-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.403470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.403450 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf" pod "5dcc7037-d1d1-4af6-9295-3e91e15c7043" (UID: "5dcc7037-d1d1-4af6-9295-3e91e15c7043"). InnerVolumeSpecName "kube-api-access-f46xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:21.403470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.403460 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm" pod "8356e552-9856-407e-b5ee-c4178e730b67" (UID: "8356e552-9856-407e-b5ee-c4178e730b67"). InnerVolumeSpecName "kube-api-access-v2fwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:21.403608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.403481 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb" pod "3fb12804-4ab5-4418-b081-cbd6a5708133" (UID: "3fb12804-4ab5-4418-b081-cbd6a5708133"). InnerVolumeSpecName "kube-api-access-kn2sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:21.502620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.502521 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\") pod \"92474479-3d19-42de-a403-9a6136d4712f\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " Nov 01 05:45:21.502810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.502728 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\") pod \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " Nov 01 05:45:21.502810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.502778 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\" (UniqueName: \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\") pod \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\" (UID: \"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc\") " Nov 01 05:45:21.502921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.502826 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\" (UniqueName: \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\") pod \"92474479-3d19-42de-a403-9a6136d4712f\" (UID: \"92474479-3d19-42de-a403-9a6136d4712f\") " Nov 01 05:45:21.502975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.502909 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca" pod "92474479-3d19-42de-a403-9a6136d4712f" (UID: "92474479-3d19-42de-a403-9a6136d4712f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:21.503170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503146 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca" pod "0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" (UID: "0a0d7b47-92c1-4cc4-b539-c54166a0ebdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:21.503361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503345 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.503440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503367 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92474479-3d19-42de-a403-9a6136d4712f-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.503440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503382 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2sb\" (UniqueName: \"kubernetes.io/projected/3fb12804-4ab5-4418-b081-cbd6a5708133-kube-api-access-kn2sb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.503440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503396 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fwm\" (UniqueName: \"kubernetes.io/projected/8356e552-9856-407e-b5ee-c4178e730b67-kube-api-access-v2fwm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.503440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.503411 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46xf\" (UniqueName: \"kubernetes.io/projected/5dcc7037-d1d1-4af6-9295-3e91e15c7043-kube-api-access-f46xf\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.505211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.505188 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j" pod "0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" (UID: "0a0d7b47-92c1-4cc4-b539-c54166a0ebdc"). InnerVolumeSpecName "kube-api-access-mtf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:21.505316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.505286 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p" pod "92474479-3d19-42de-a403-9a6136d4712f" (UID: "92474479-3d19-42de-a403-9a6136d4712f"). InnerVolumeSpecName "kube-api-access-mx86p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:21.605246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.605208 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtf8j\" (UniqueName: \"kubernetes.io/projected/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc-kube-api-access-mtf8j\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:21.605246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:21.605243 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx86p\" (UniqueName: \"kubernetes.io/projected/92474479-3d19-42de-a403-9a6136d4712f-kube-api-access-mx86p\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:22.059841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.059802 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" event={"ID":"0a0d7b47-92c1-4cc4-b539-c54166a0ebdc","Type":"ContainerDied","Data":"2aa7f2d762f10398a5b24d1cbe458496704de97bd8f9f627474f76636e8bbd51"} Nov 01 05:45:22.060043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.059869 4425 scope.go:122] "RemoveContainer" containerID="318446af23ffc52f6cc3e34013c609c64cf998860d28dfa78a787da8b7a76fae" Nov 01 05:45:22.060043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.059810 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt" Nov 01 05:45:22.061464 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.061443 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" Nov 01 05:45:22.061560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.061446 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8" event={"ID":"3fb12804-4ab5-4418-b081-cbd6a5708133","Type":"ContainerDied","Data":"f6941642d6ed89c40678133bab2c035fc3132dbf6a942f9961ef41f7fffc240a"} Nov 01 05:45:22.062926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.062910 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" Nov 01 05:45:22.063046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.062946 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5" event={"ID":"8356e552-9856-407e-b5ee-c4178e730b67","Type":"ContainerDied","Data":"59d09e9429a33c90967df546b7196882747ab8b8c91989b9f826c62ab6fa23a1"} Nov 01 05:45:22.064565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.064545 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" Nov 01 05:45:22.064670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.064624 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh" event={"ID":"5dcc7037-d1d1-4af6-9295-3e91e15c7043","Type":"ContainerDied","Data":"49ad5fa7fe34ca5e8778e1362566b5622681d8de1e1e4ec070f9c343a612aad5"} Nov 01 05:45:22.068843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.068813 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" event={"ID":"92474479-3d19-42de-a403-9a6136d4712f","Type":"ContainerDied","Data":"1b937e3f17e1be1ba1410a320496611977aeac8312c09125a759e80dd679c360"} Nov 01 05:45:22.068965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.068855 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b" Nov 01 05:45:22.072221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.072198 4425 scope.go:122] "RemoveContainer" containerID="31b3e0fd8489437cbd195a315d4c9bffdc5d132c38b556129c35924ce4601c82" Nov 01 05:45:22.087434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.087413 4425 scope.go:122] "RemoveContainer" containerID="59c88add8afeb8edc180d73a9f44f8fe961132952c7128d234f88c0e9dcb286e" Nov 01 05:45:22.099275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.099239 4425 scope.go:122] "RemoveContainer" containerID="5a674834e2886fad27b0a607c1cdf20d411938c1b279b51951cccefaee65ff13" Nov 01 05:45:22.109550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.109531 4425 scope.go:122] "RemoveContainer" containerID="ba9bbe11971b1f8f82dfad2b80006d4b8c708d41c69da3f1e500d6375130f8fc" Nov 01 05:45:22.428793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.428596 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:22.432977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.432947 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891865-lp8m5"] Nov 01 05:45:22.544516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.544488 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:22.547963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.547926 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-faa8ff-29891865-5twf8"] Nov 01 05:45:22.658856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.658811 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:22.668345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.668314 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8815d2-29891865-nx29b"] Nov 01 05:45:22.779492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.779395 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:22.782650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.782630 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2fd4c9-29891865-5jqvt"] Nov 01 05:45:22.896470 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.896430 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:22.903578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.903551 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d70ff-29891865-ljbhh"] Nov 01 05:45:22.978580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.978546 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" path="/var/lib/kubelet/pods/0a0d7b47-92c1-4cc4-b539-c54166a0ebdc/volumes" Nov 01 05:45:22.979122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.979103 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" path="/var/lib/kubelet/pods/2a137f1f-902a-44ed-9840-63fbfc34aa25/volumes" Nov 01 05:45:22.979518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.979503 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" path="/var/lib/kubelet/pods/3fb12804-4ab5-4418-b081-cbd6a5708133/volumes" Nov 01 05:45:22.980158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.980143 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" path="/var/lib/kubelet/pods/5dcc7037-d1d1-4af6-9295-3e91e15c7043/volumes" Nov 01 05:45:22.980565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.980551 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8356e552-9856-407e-b5ee-c4178e730b67" path="/var/lib/kubelet/pods/8356e552-9856-407e-b5ee-c4178e730b67/volumes" Nov 01 05:45:22.980908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:22.980895 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92474479-3d19-42de-a403-9a6136d4712f" path="/var/lib/kubelet/pods/92474479-3d19-42de-a403-9a6136d4712f/volumes" Nov 01 05:45:23.491335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.491288 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" podUID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" containerName="payload-processing" containerID="cri-o://b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e" gracePeriod=25 Nov 01 05:45:23.599996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.599958 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" podUID="872cce58-de80-4040-a73a-43f828fac090" containerName="payload-pre-processing" containerID="cri-o://7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf" gracePeriod=25 Nov 01 05:45:23.775551 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.775523 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:45:23.886175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.886150 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:45:23.932370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.932331 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\") pod \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " Nov 01 05:45:23.932559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.932486 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\" (UniqueName: \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\") pod \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\" (UID: \"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f\") " Nov 01 05:45:23.932771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.932750 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume" pod "5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" (UID: "5bb04c9c-5aa3-4b56-93d5-fa15e06a488f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:23.932978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.932956 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:23.934693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:23.934671 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m" pod "5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" (UID: "5bb04c9c-5aa3-4b56-93d5-fa15e06a488f"). InnerVolumeSpecName "kube-api-access-pps7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:24.033623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.033535 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\" (UniqueName: \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\") pod \"872cce58-de80-4040-a73a-43f828fac090\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " Nov 01 05:45:24.033623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.033615 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\") pod \"872cce58-de80-4040-a73a-43f828fac090\" (UID: \"872cce58-de80-4040-a73a-43f828fac090\") " Nov 01 05:45:24.033989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.033960 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume" pod "872cce58-de80-4040-a73a-43f828fac090" (UID: "872cce58-de80-4040-a73a-43f828fac090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:24.034078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.034059 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pps7m\" (UniqueName: \"kubernetes.io/projected/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f-kube-api-access-pps7m\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:24.034078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.034075 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872cce58-de80-4040-a73a-43f828fac090-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:24.035967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.035934 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds" pod "872cce58-de80-4040-a73a-43f828fac090" (UID: "872cce58-de80-4040-a73a-43f828fac090"). InnerVolumeSpecName "kube-api-access-s8mds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:24.097351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.097310 4425 generic.go:356] "Generic (PLEG): container finished" podID="872cce58-de80-4040-a73a-43f828fac090" containerID="7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf" exitCode=0 Nov 01 05:45:24.097516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.097362 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" Nov 01 05:45:24.097516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.097394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" event={"ID":"872cce58-de80-4040-a73a-43f828fac090","Type":"ContainerDied","Data":"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf"} Nov 01 05:45:24.097516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.097473 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf" event={"ID":"872cce58-de80-4040-a73a-43f828fac090","Type":"ContainerDied","Data":"58fc6d170e803f83454f019801b03b02d9c0dc9c769112eaa13ee3b8f8d1f194"} Nov 01 05:45:24.097516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.097497 4425 scope.go:122] "RemoveContainer" containerID="7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf" Nov 01 05:45:24.099191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.099169 4425 generic.go:356] "Generic (PLEG): container finished" podID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" containerID="b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e" exitCode=0 Nov 01 05:45:24.099323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.099234 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" Nov 01 05:45:24.099323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.099283 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" event={"ID":"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f","Type":"ContainerDied","Data":"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e"} Nov 01 05:45:24.099323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.099322 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq" event={"ID":"5bb04c9c-5aa3-4b56-93d5-fa15e06a488f","Type":"ContainerDied","Data":"3af9970e0e56d387adde38439776f4a6b08be8a87ff2dc6a81cfb823aff6b0cb"} Nov 01 05:45:24.108240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.108069 4425 scope.go:122] "RemoveContainer" containerID="7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf" Nov 01 05:45:24.108420 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:24.108394 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf\": container with ID starting with 7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf not found: ID does not exist" containerID="7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf" Nov 01 05:45:24.108565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.108427 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf"} err="failed to get container status \"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf\": rpc error: code = NotFound desc = could not find container \"7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf\": container with ID starting with 7368de3f2c47f5ef424c11351918cfc274eda8d6f588623aee810decba217edf not found: ID does not exist" Nov 01 05:45:24.108565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.108450 4425 scope.go:122] "RemoveContainer" containerID="b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e" Nov 01 05:45:24.117852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.117833 4425 scope.go:122] "RemoveContainer" containerID="b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e" Nov 01 05:45:24.118102 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:24.118083 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e\": container with ID starting with b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e not found: ID does not exist" containerID="b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e" Nov 01 05:45:24.118202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.118109 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e"} err="failed to get container status \"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e\": rpc error: code = NotFound desc = could not find container \"b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e\": container with ID starting with b50eba7c084b895f7c49d2ab3945306251428959ea1a5135227cfcad75cca16e not found: ID does not exist" Nov 01 05:45:24.134878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.134853 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mds\" (UniqueName: \"kubernetes.io/projected/872cce58-de80-4040-a73a-43f828fac090-kube-api-access-s8mds\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:24.265518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.265487 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:45:24.269763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.269737 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-faa8ff-d4bb67df9-zd2sq"] Nov 01 05:45:24.380173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.380093 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:45:24.384124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.384099 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-faa8ff-5b96b764b7-l9wrf"] Nov 01 05:45:24.973580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.973538 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" path="/var/lib/kubelet/pods/5bb04c9c-5aa3-4b56-93d5-fa15e06a488f/volumes" Nov 01 05:45:24.974125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:24.974103 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872cce58-de80-4040-a73a-43f828fac090" path="/var/lib/kubelet/pods/872cce58-de80-4040-a73a-43f828fac090/volumes" Nov 01 05:45:26.949833 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:26.949792 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:45:28.978310 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:28.978247 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:31.053989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:31.053959 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:45:33.325478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.325448 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh"] Nov 01 05:45:33.325873 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:33.325766 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" podUID="dcfcbe84-496c-46b4-a708-aec6bde4700e" Nov 01 05:45:33.358110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358075 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:45:33.358667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358654 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358668 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358682 4425 cpu_manager.go:414] "containerMap: removing container" podUID="872cce58-de80-4040-a73a-43f828fac090" containerName="payload-pre-processing" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358688 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872cce58-de80-4040-a73a-43f828fac090" containerName="payload-pre-processing" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358697 4425 cpu_manager.go:414] "containerMap: removing container" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358701 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358711 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358716 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.358723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358722 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" containerName="payload-processing" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358727 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" containerName="payload-processing" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358737 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358741 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358752 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358758 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358765 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358769 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358777 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358781 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358791 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358795 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358806 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358810 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358821 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358825 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358831 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358836 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358842 4425 cpu_manager.go:414] "containerMap: removing container" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358846 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358854 4425 cpu_manager.go:414] "containerMap: removing container" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358858 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358865 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358869 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358878 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358882 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358888 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358892 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358903 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358908 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358916 4425 cpu_manager.go:414] "containerMap: removing container" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358921 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358928 4425 cpu_manager.go:414] "containerMap: removing container" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.358977 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.358932 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359016 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359025 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359032 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069f400-066e-4467-8855-6f97713c7470" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359038 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359045 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359050 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359055 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359060 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359066 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359071 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359075 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359080 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb12804-4ab5-4418-b081-cbd6a5708133" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359084 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f219aa5-6c8b-458b-9918-f309fd517260" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359089 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0d7b47-92c1-4cc4-b539-c54166a0ebdc" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359094 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="92474479-3d19-42de-a403-9a6136d4712f" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359101 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="872cce58-de80-4040-a73a-43f828fac090" containerName="payload-pre-processing" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359106 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359110 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359115 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb04c9c-5aa3-4b56-93d5-fa15e06a488f" containerName="payload-processing" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359121 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359128 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:33.359917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.359132 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:33.362605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.362588 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.415832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.415791 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-30604247b165\"" Nov 01 05:45:33.436495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.436463 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:45:33.448822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.448786 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.448999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.448930 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk26z\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.448999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.448994 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.449132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.449028 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.550280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.550215 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk26z\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.550280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.550269 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.550521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.550304 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.550521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.550422 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.551064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.551035 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.551180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.551104 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.553023 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.553004 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.612504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.612429 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk26z\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:33.676348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:33.676312 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:34.059151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.059122 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:45:34.059397 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:34.059373 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51df0dd7_b2fb_4b59_a0cd_49e7a1c47a3e.slice/crio-d51a3b36bd0ff0605f37076332e248186bff8a39ecd992802aa55ab8bc97697a WatchSource:0}: Error finding container d51a3b36bd0ff0605f37076332e248186bff8a39ecd992802aa55ab8bc97697a: Status 404 returned error can't find the container with id d51a3b36bd0ff0605f37076332e248186bff8a39ecd992802aa55ab8bc97697a Nov 01 05:45:34.173904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.173858 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" event={"ID":"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e","Type":"ContainerStarted","Data":"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2"} Nov 01 05:45:34.174078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.173938 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:45:34.174078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.174042 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:34.174200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.174077 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" event={"ID":"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e","Type":"ContainerStarted","Data":"d51a3b36bd0ff0605f37076332e248186bff8a39ecd992802aa55ab8bc97697a"} Nov 01 05:45:34.181510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.181485 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:45:34.259507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.259470 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\") pod \"dcfcbe84-496c-46b4-a708-aec6bde4700e\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " Nov 01 05:45:34.259507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.259509 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\") pod \"dcfcbe84-496c-46b4-a708-aec6bde4700e\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " Nov 01 05:45:34.259770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.259623 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\") pod \"dcfcbe84-496c-46b4-a708-aec6bde4700e\" (UID: \"dcfcbe84-496c-46b4-a708-aec6bde4700e\") " Nov 01 05:45:34.259981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.259953 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume" pod "dcfcbe84-496c-46b4-a708-aec6bde4700e" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:34.260139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.260032 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume" pod "dcfcbe84-496c-46b4-a708-aec6bde4700e" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:34.262218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.262197 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn" pod "dcfcbe84-496c-46b4-a708-aec6bde4700e" (UID: "dcfcbe84-496c-46b4-a708-aec6bde4700e"). InnerVolumeSpecName "kube-api-access-wdpwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:34.295247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.295194 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" podStartSLOduration=1.295179347 podStartE2EDuration="1.295179347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:34.292131875 +0000 UTC m=+2549.432720794" watchObservedRunningTime="2026-11-01 05:45:34.295179347 +0000 UTC m=+2549.435768265" Nov 01 05:45:34.361171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.361134 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpwn\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-kube-api-access-wdpwn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:34.361171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.361165 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:34.361171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:34.361175 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfcbe84-496c-46b4-a708-aec6bde4700e-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:35.180994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:35.180957 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh" Nov 01 05:45:35.413021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:35.412981 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh"] Nov 01 05:45:35.416735 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:35.416707 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576c69zh"] Nov 01 05:45:35.475166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:35.475132 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcfcbe84-496c-46b4-a708-aec6bde4700e-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:36.974819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:36.974783 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfcbe84-496c-46b4-a708-aec6bde4700e" path="/var/lib/kubelet/pods/dcfcbe84-496c-46b4-a708-aec6bde4700e/volumes" Nov 01 05:45:38.969198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:38.969145 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:45.184230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:45.184201 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:45:45.356699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:45.356667 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:45:45.358065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:45.358035 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:45:48.641330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641236 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:45:48.641855 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641840 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:48.641901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641856 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:48.641901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641870 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:48.641901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641874 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356e552-9856-407e-b5ee-c4178e730b67" containerName="cleanup" Nov 01 05:45:48.641998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641927 4425 cpu_manager.go:414] "containerMap: removing container" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:48.641998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.641933 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:48.642069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.642017 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a137f1f-902a-44ed-9840-63fbfc34aa25" containerName="cleanup" Nov 01 05:45:48.642069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.642029 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcc7037-d1d1-4af6-9295-3e91e15c7043" containerName="cleanup" Nov 01 05:45:48.642069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.642039 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0be2c-7ffc-4892-9b04-c7454b8a780e" containerName="cleanup" Nov 01 05:45:48.647141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.647118 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.708468 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.708433 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:45:48.750725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.750682 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25565930\"/\"kube-root-ca.crt\"" Nov 01 05:45:48.750945 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.750880 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25565930\"/\"e2e-lifecycle-model-25565930-kserve-self-signed-certs\"" Nov 01 05:45:48.751015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.750939 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25565930\"/\"openshift-service-ca.crt\"" Nov 01 05:45:48.751015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.750968 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25565930\"/\"default-dockercfg-2cpdz\"" Nov 01 05:45:48.841338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841302 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.841564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841348 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.841564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841512 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.841722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841561 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6snn\" (UniqueName: \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.841722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841645 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.841844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.841752 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.942924 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.942976 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943025 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943048 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6snn\" (UniqueName: \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943074 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943247 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943147 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943400 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943451 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.943538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.943512 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.945611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.945585 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:48.945933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:48.945914 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:49.005389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:49.005356 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6snn\" (UniqueName: \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\") pod \"e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:49.260555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:49.260459 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:49.667635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:49.664781 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:45:50.293476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.293439 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" event={"ID":"36ad069d-f15c-4882-9029-f5b1eb93baf4","Type":"ContainerStarted","Data":"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee"} Nov 01 05:45:50.293681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.293486 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" event={"ID":"36ad069d-f15c-4882-9029-f5b1eb93baf4","Type":"ContainerStarted","Data":"bbd4690fd35d6dddc78d88875d736ad4143906b65b7ca94855a9f7d3b9debd88"} Nov 01 05:45:50.357830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.357749 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" containerID="cri-o://e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5" gracePeriod=25 Nov 01 05:45:50.413524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.413447 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" podStartSLOduration=2.413427007 podStartE2EDuration="2.413427007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:50.412550127 +0000 UTC m=+2565.553139065" watchObservedRunningTime="2026-11-01 05:45:50.413427007 +0000 UTC m=+2565.554015993" Nov 01 05:45:50.634876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.634839 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:45:50.738168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738132 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\") pod \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " Nov 01 05:45:50.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738272 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\") pod \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " Nov 01 05:45:50.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738338 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\") pod \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " Nov 01 05:45:50.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738412 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\") pod \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\" (UID: \"7801c258-cc25-42ba-baf4-6cdf0b07d67c\") " Nov 01 05:45:50.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738569 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume" pod "7801c258-cc25-42ba-baf4-6cdf0b07d67c" (UID: "7801c258-cc25-42ba-baf4-6cdf0b07d67c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:50.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738632 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume" pod "7801c258-cc25-42ba-baf4-6cdf0b07d67c" (UID: "7801c258-cc25-42ba-baf4-6cdf0b07d67c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:45:50.739014 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.738986 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:50.739014 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.739002 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7801c258-cc25-42ba-baf4-6cdf0b07d67c-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:50.740806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.740779 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials" pod "7801c258-cc25-42ba-baf4-6cdf0b07d67c" (UID: "7801c258-cc25-42ba-baf4-6cdf0b07d67c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:50.741196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.741177 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf" pod "7801c258-cc25-42ba-baf4-6cdf0b07d67c" (UID: "7801c258-cc25-42ba-baf4-6cdf0b07d67c"). InnerVolumeSpecName "kube-api-access-dlsjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:45:50.839915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.839876 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:50.839915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:50.839909 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsjf\" (UniqueName: \"kubernetes.io/projected/7801c258-cc25-42ba-baf4-6cdf0b07d67c-kube-api-access-dlsjf\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:45:51.261526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.261439 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:51.303786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.303753 4425 generic.go:356] "Generic (PLEG): container finished" podID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerID="e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5" exitCode=0 Nov 01 05:45:51.303967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.303808 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" Nov 01 05:45:51.303967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.303839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" event={"ID":"7801c258-cc25-42ba-baf4-6cdf0b07d67c","Type":"ContainerDied","Data":"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5"} Nov 01 05:45:51.303967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.303921 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml" event={"ID":"7801c258-cc25-42ba-baf4-6cdf0b07d67c","Type":"ContainerDied","Data":"d02327afc71729b56c7178b9852b010bb9a91768f411973671618d7211b65388"} Nov 01 05:45:51.303967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.303962 4425 scope.go:122] "RemoveContainer" containerID="e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5" Nov 01 05:45:51.317058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.317034 4425 scope.go:122] "RemoveContainer" containerID="e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5" Nov 01 05:45:51.317443 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:51.317420 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5\": container with ID starting with e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5 not found: ID does not exist" containerID="e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5" Nov 01 05:45:51.317524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.317451 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5"} err="failed to get container status \"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5\": rpc error: code = NotFound desc = could not find container \"e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5\": container with ID starting with e36f1eb7442ff5d4b49327de17e60da0122ef08905dbdb540de5077d66aa14f5 not found: ID does not exist" Nov 01 05:45:51.477401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.477354 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:45:51.525642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:51.525560 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f8549dsml"] Nov 01 05:45:52.978908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:52.978869 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" path="/var/lib/kubelet/pods/7801c258-cc25-42ba-baf4-6cdf0b07d67c/volumes" Nov 01 05:45:53.225056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.225019 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 05:45:53.225678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.225663 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" Nov 01 05:45:53.225736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.225681 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" Nov 01 05:45:53.225844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.225835 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7801c258-cc25-42ba-baf4-6cdf0b07d67c" containerName="payload-processing" Nov 01 05:45:53.231086 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.231034 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.276538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.276504 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 05:45:53.280893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.280870 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.281988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.281958 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 05:45:53.287304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.287278 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.323059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.323026 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\"" Nov 01 05:45:53.329564 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.329531 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 05:45:53.371199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371162 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.371433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371247 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.371433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371335 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxtrb\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.371433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371405 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.371433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371429 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.371696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371449 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.371696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371472 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.371696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371579 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.371696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371604 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shk9\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.371696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371633 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.371985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371699 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.371985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.371730 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.372236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372209 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.372371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372316 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.372371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372351 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.372588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372554 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.372661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372614 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.372732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372684 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.372794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372728 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.372794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372765 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.372794 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372790 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.372964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372824 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.372964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.372964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372914 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859ff\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.372964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372942 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.373691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.372973 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.373691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.373016 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.404942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.404867 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 05:45:53.410548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.410512 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 05:45:53.474221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474176 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474215 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474308 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474348 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474373 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474397 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474421 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474453 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474492 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859ff\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474518 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474542 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474592 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474643 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474727 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474753 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxtrb\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474794 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474824 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474851 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474885 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.474972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474951 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474981 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shk9\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.474992 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475018 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475052 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475087 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475117 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475152 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475187 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475282 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.475619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475520 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.476170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475738 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.476170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.475985 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.476170 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.476148 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.476478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.476443 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.476792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.476753 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.477550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.477523 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.478371 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.478342 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.478665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.478644 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.478799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.478772 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.479373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.479069 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.479487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.479377 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.480011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.479948 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.480312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.480283 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.480707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.480663 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.480805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.480708 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.481620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.481554 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.481897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.481878 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.483391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.483363 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.536789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.536751 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.539776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.539744 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.540172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.540143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.540649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.540627 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859ff\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\") pod \"maas-default-gateway-openshift-default-58d898bd55-mpdzz\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.546379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.546340 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:53.558549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.558524 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shk9\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\") pod \"maas-default-gateway-openshift-default-58d898bd55-sd9zb\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.558676 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.558624 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxtrb\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\") pod \"maas-default-gateway-openshift-default-58d898bd55-ztp4p\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.596862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.596830 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:53.606933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.606713 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:53.984934 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:53.984904 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d297bba_2e2e_4cad_9773_27cfb3b395f7.slice/crio-c11f95b89aecb4de27cffc063917dbb17c459107452b0200519aa13204aa5257 WatchSource:0}: Error finding container c11f95b89aecb4de27cffc063917dbb17c459107452b0200519aa13204aa5257: Status 404 returned error can't find the container with id c11f95b89aecb4de27cffc063917dbb17c459107452b0200519aa13204aa5257 Nov 01 05:45:53.985429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.985348 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 05:45:53.987790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.987746 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:53.987892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.987822 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:53.987892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:53.987852 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.038447 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:54.038412 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f3588d_5397_4aa5_b613_6b4fdf41bf52.slice/crio-9de5d2b1c7ffa9e366390a49a93b2087c0189933d1ab8ce5d0e815d357a02817 WatchSource:0}: Error finding container 9de5d2b1c7ffa9e366390a49a93b2087c0189933d1ab8ce5d0e815d357a02817: Status 404 returned error can't find the container with id 9de5d2b1c7ffa9e366390a49a93b2087c0189933d1ab8ce5d0e815d357a02817 Nov 01 05:45:54.038633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.038608 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 05:45:54.044408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.041941 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.044408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.042035 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.044408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.042085 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.087706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.087675 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 05:45:54.088633 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:45:54.088607 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8787e714_bc21_4665_b904_6c521c49244e.slice/crio-9b4358e81716cdcfe59d4db48659e0bdb262fb1f25706121939736cb962debe2 WatchSource:0}: Error finding container 9b4358e81716cdcfe59d4db48659e0bdb262fb1f25706121939736cb962debe2: Status 404 returned error can't find the container with id 9b4358e81716cdcfe59d4db48659e0bdb262fb1f25706121939736cb962debe2 Nov 01 05:45:54.091339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.091306 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.091451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.091376 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.091451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.091411 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:45:54.336279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.335791 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" event={"ID":"8787e714-bc21-4665-b904-6c521c49244e","Type":"ContainerStarted","Data":"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876"} Nov 01 05:45:54.336279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.335863 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" event={"ID":"8787e714-bc21-4665-b904-6c521c49244e","Type":"ContainerStarted","Data":"9b4358e81716cdcfe59d4db48659e0bdb262fb1f25706121939736cb962debe2"} Nov 01 05:45:54.342149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.338173 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" event={"ID":"8d297bba-2e2e-4cad-9773-27cfb3b395f7","Type":"ContainerStarted","Data":"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5"} Nov 01 05:45:54.342149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.338228 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" event={"ID":"8d297bba-2e2e-4cad-9773-27cfb3b395f7","Type":"ContainerStarted","Data":"c11f95b89aecb4de27cffc063917dbb17c459107452b0200519aa13204aa5257"} Nov 01 05:45:54.342149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.340927 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" event={"ID":"d2f3588d-5397-4aa5-b613-6b4fdf41bf52","Type":"ContainerStarted","Data":"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd"} Nov 01 05:45:54.342149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.340978 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" event={"ID":"d2f3588d-5397-4aa5-b613-6b4fdf41bf52","Type":"ContainerStarted","Data":"9de5d2b1c7ffa9e366390a49a93b2087c0189933d1ab8ce5d0e815d357a02817"} Nov 01 05:45:54.475613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.475541 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podStartSLOduration=1.475515698 podStartE2EDuration="1.475515698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:54.471337131 +0000 UTC m=+2569.611926076" watchObservedRunningTime="2026-11-01 05:45:54.475515698 +0000 UTC m=+2569.616104684" Nov 01 05:45:54.546801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.546755 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:45:54.597448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.597356 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:45:54.606955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.606888 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podStartSLOduration=1.6068672240000001 podStartE2EDuration="1.606867224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:54.604086586 +0000 UTC m=+2569.744675506" watchObservedRunningTime="2026-11-01 05:45:54.606867224 +0000 UTC m=+2569.747456141" Nov 01 05:45:54.610294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.610223 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:45:54.730195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:54.730141 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podStartSLOduration=1.73012425 podStartE2EDuration="1.73012425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:54.727009208 +0000 UTC m=+2569.867598148" watchObservedRunningTime="2026-11-01 05:45:54.73012425 +0000 UTC m=+2569.870713233" Nov 01 05:45:55.557279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.553237 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:45:55.557279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.553332 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:45:55.612579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.612462 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:45:55.612579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.612516 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:45:55.614600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.614571 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:45:55.614834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:55.614807 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:45:56.571012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.570974 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:45:56.571663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.571630 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:45:56.597554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.597516 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:45:56.597858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.597830 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:45:56.611731 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.611692 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:45:56.611996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:56.611969 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:45:57.019995 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:45:57.019898 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:45:57.551280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.550675 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:45:57.551280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.550739 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:45:57.614294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.604375 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:45:57.614294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.604439 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:45:57.625942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.625901 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:45:57.631399 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:57.631352 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:45:58.069281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.061749 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw"] Nov 01 05:45:58.119595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.117411 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30193dbb-qwdxw"] Nov 01 05:45:58.553575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.550683 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:45:58.553575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.550751 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:45:58.605282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.603220 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:45:58.605282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.603303 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:45:58.620479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.620436 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:45:58.621385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:58.621327 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:45:59.021278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.020746 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd5024b-02b4-4e95-b2d7-16725bfb9ef8" path="/var/lib/kubelet/pods/7cd5024b-02b4-4e95-b2d7-16725bfb9ef8/volumes" Nov 01 05:45:59.276340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.272975 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:59.328662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.328636 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:59.532819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.532739 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:45:59.563458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.563409 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:45:59.563775 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.563743 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:45:59.608991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.608954 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:45:59.623478 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.623422 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:45:59.624712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.624664 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:45:59.624914 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:45:59.624733 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:46:00.547797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.547758 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:46:00.548036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.547846 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:46:00.597514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.597476 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:46:00.597725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.597542 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:46:00.611335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.611293 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:46:00.611529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:00.611357 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:46:01.030052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.030002 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:46:01.030717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.030321 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" containerID="cri-o://e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf" gracePeriod=30 Nov 01 05:46:01.547377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.547333 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 01 05:46:01.547584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.547424 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 01 05:46:01.597692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.597645 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 01 05:46:01.597920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.597759 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 01 05:46:01.611073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.611032 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-sd9zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Nov 01 05:46:01.611317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.611110 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Nov 01 05:46:01.751500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.751456 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt"] Nov 01 05:46:01.758598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.758568 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:01.813147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.812150 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:46:01.813147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.812330 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:46:01.813147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.813001 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:46:01.813147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.813115 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:46:01.813574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.813315 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt"] Nov 01 05:46:01.899521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.899465 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:01.899743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.899660 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:01.899827 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:01.899809 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgbl\" (UniqueName: \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.001160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.001083 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.001402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.001321 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgbl\" (UniqueName: \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.001506 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.001475 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.002102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.002061 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.004225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.004193 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.064102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.064001 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgbl\" (UniqueName: \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\") pod \"maas-api-revoke-sub-bb15465b-frpqt\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.075749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.075704 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:02.479981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.479945 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt"] Nov 01 05:46:02.561028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:02.560879 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" event={"ID":"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7","Type":"ContainerStarted","Data":"883dae1ecc25ab15e56fb3edf9152439a7eda77ec475d4e3817fa3839a7ba4f5"} Nov 01 05:46:03.503670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.503634 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:46:03.504577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.504513 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:46:03.506211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.506186 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 05:46:03.510753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.510726 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:46:03.512083 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.512058 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:46:03.518327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.517469 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 05:46:03.522949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.521458 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:46:03.522949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.522817 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:46:03.523728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.523707 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 05:46:03.574583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.574544 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" event={"ID":"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7","Type":"ContainerStarted","Data":"adff3bfa7d9f42f0388dfd915a9ac25043b83c1071bfab9b78ca661f27ae8840"} Nov 01 05:46:03.796525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:03.796418 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" podStartSLOduration=2.7964036549999998 podStartE2EDuration="2.796403655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:03.793756681 +0000 UTC m=+2578.934345602" watchObservedRunningTime="2026-11-01 05:46:03.796403655 +0000 UTC m=+2578.936992574" Nov 01 05:46:04.583817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:04.583776 4425 generic.go:356] "Generic (PLEG): container finished" podID="fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" containerID="adff3bfa7d9f42f0388dfd915a9ac25043b83c1071bfab9b78ca661f27ae8840" exitCode=0 Nov 01 05:46:04.584297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:04.583864 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" event={"ID":"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7","Type":"ContainerDied","Data":"adff3bfa7d9f42f0388dfd915a9ac25043b83c1071bfab9b78ca661f27ae8840"} Nov 01 05:46:05.759992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.759956 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:05.822139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.822102 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\") pod \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " Nov 01 05:46:05.822411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.822157 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\") pod \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " Nov 01 05:46:05.822411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.822291 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\" (UniqueName: \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\") pod \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\" (UID: \"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7\") " Nov 01 05:46:05.822613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.822580 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca" pod "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" (UID: "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:05.822951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.822895 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:05.824697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.824665 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca" pod "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" (UID: "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:05.824804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.824695 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl" pod "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" (UID: "fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7"). InnerVolumeSpecName "kube-api-access-qxgbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:05.923908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.923823 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:05.923908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:05.923856 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgbl\" (UniqueName: \"kubernetes.io/projected/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7-kube-api-access-qxgbl\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.292106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.292081 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:46:06.328438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328407 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328488 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328519 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328576 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328677 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328718 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328798 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328777 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328864 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328826 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.328929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328887 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:06.328990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328927 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:06.328990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.328943 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\") pod \"990e9ab6-e3b5-4ead-a448-4fc047650186\" (UID: \"990e9ab6-e3b5-4ead-a448-4fc047650186\") " Nov 01 05:46:06.329115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329051 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:06.329179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329079 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:06.329179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329165 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:06.329739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329717 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.329814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329746 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990e9ab6-e3b5-4ead-a448-4fc047650186-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.329814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329763 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.329814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329779 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.329814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.329792 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.330926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.330905 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:06.331120 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.331101 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "kube-api-access-9nwv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:06.331435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.331417 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:46:06.331485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.331458 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy" pod "990e9ab6-e3b5-4ead-a448-4fc047650186" (UID: "990e9ab6-e3b5-4ead-a448-4fc047650186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:46:06.431546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.431507 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.431546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.431539 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.431546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.431552 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwv7\" (UniqueName: \"kubernetes.io/projected/990e9ab6-e3b5-4ead-a448-4fc047650186-kube-api-access-9nwv7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.431799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.431564 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990e9ab6-e3b5-4ead-a448-4fc047650186-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:06.600058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.600020 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" Nov 01 05:46:06.600058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.600034 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt" event={"ID":"fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7","Type":"ContainerDied","Data":"883dae1ecc25ab15e56fb3edf9152439a7eda77ec475d4e3817fa3839a7ba4f5"} Nov 01 05:46:06.600387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.600091 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883dae1ecc25ab15e56fb3edf9152439a7eda77ec475d4e3817fa3839a7ba4f5" Nov 01 05:46:06.602225 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.602197 4425 generic.go:356] "Generic (PLEG): container finished" podID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerID="e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf" exitCode=0 Nov 01 05:46:06.602421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.602268 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" Nov 01 05:46:06.602421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.602282 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerDied","Data":"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf"} Nov 01 05:46:06.602421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.602336 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc" event={"ID":"990e9ab6-e3b5-4ead-a448-4fc047650186","Type":"ContainerDied","Data":"f39f1eb5ff4bf45b27e611513c228be26872ce2c28868a42a0c7065b13177331"} Nov 01 05:46:06.602421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.602371 4425 scope.go:122] "RemoveContainer" containerID="e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf" Nov 01 05:46:06.614475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.614144 4425 scope.go:122] "RemoveContainer" containerID="6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2" Nov 01 05:46:06.625507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.625489 4425 scope.go:122] "RemoveContainer" containerID="e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf" Nov 01 05:46:06.625827 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:06.625799 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf\": container with ID starting with e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf not found: ID does not exist" containerID="e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf" Nov 01 05:46:06.625926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.625832 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf"} err="failed to get container status \"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf\": rpc error: code = NotFound desc = could not find container \"e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf\": container with ID starting with e10f2d3464c2bcb3c236861491c4a1ec4f75e13e48dd586b4802a0fa413488cf not found: ID does not exist" Nov 01 05:46:06.625926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.625849 4425 scope.go:122] "RemoveContainer" containerID="6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2" Nov 01 05:46:06.626116 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:06.626092 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2\": container with ID starting with 6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2 not found: ID does not exist" containerID="6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2" Nov 01 05:46:06.626171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.626127 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2"} err="failed to get container status \"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2\": rpc error: code = NotFound desc = could not find container \"6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2\": container with ID starting with 6554383c56d3ac9acbcf5f52bdfbbbbb84fb0850e266816a862a3d55d4d79ea2 not found: ID does not exist" Nov 01 05:46:06.823986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.823951 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:46:06.830164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.830140 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-faa8ff-openshift-default-667f695f57-982wc"] Nov 01 05:46:06.975436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.975355 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" path="/var/lib/kubelet/pods/990e9ab6-e3b5-4ead-a448-4fc047650186/volumes" Nov 01 05:46:06.987924 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.987896 4425 scope.go:122] "RemoveContainer" containerID="7ac86a166267c9135dc1fbc9011963f32d806833da4fb355660b9c8eb1dad7c5" Nov 01 05:46:06.998058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:06.998034 4425 scope.go:122] "RemoveContainer" containerID="de93cd8fc643b73ee756f80d2f4543c5fdb3e71bcbf060a98ac1274af9471889" Nov 01 05:46:07.007697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:07.007677 4425 scope.go:122] "RemoveContainer" containerID="5b64c569609186243d689193f539dc034457a7b919026d53920064e75b8ab0cf" Nov 01 05:46:08.631961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:08.631929 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:46:10.048161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:10.048129 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb"] Nov 01 05:46:10.098859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:10.098828 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77965773-sklzb"] Nov 01 05:46:10.974745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:10.974713 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6387c9a-c3d7-4374-bb07-f3bac5e2387e" path="/var/lib/kubelet/pods/f6387c9a-c3d7-4374-bb07-f3bac5e2387e/volumes" Nov 01 05:46:13.614602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.614560 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7"] Nov 01 05:46:13.615418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615396 4425 cpu_manager.go:414] "containerMap: removing container" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:13.615418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615419 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:13.615603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615464 4425 cpu_manager.go:414] "containerMap: removing container" podUID="fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" containerName="revoke-keys" Nov 01 05:46:13.615603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615471 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" containerName="revoke-keys" Nov 01 05:46:13.615603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615494 4425 cpu_manager.go:414] "containerMap: removing container" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:13.615603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615502 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:13.615815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615639 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" containerName="revoke-keys" Nov 01 05:46:13.615815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.615654 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:13.621406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.621384 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.632648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.632616 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" podUID="b768b926-d65b-4ed7-afc0-bf5111096264" containerName="payload-pre-processing" containerID="cri-o://4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9" gracePeriod=25 Nov 01 05:46:13.677729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.677698 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7"] Nov 01 05:46:13.723372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.723342 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:46:13.723372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.723363 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:46:13.723593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.723373 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:46:13.723593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.723436 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:46:13.726895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.726875 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.727105 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.727088 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.727168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.727151 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx2k\" (UniqueName: \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.828669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.828633 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.828823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.828712 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx2k\" (UniqueName: \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.828823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.828772 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.829453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.829426 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.831233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.831209 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.892089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.892011 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx2k\" (UniqueName: \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\") pod \"maas-api-revoke-keys-e2e-derive-31e728ff-4skm7\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.935422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.935396 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:13.936707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:13.936686 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:46:14.031265 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.031225 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\" (UniqueName: \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\") pod \"b768b926-d65b-4ed7-afc0-bf5111096264\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " Nov 01 05:46:14.031406 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.031374 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\") pod \"b768b926-d65b-4ed7-afc0-bf5111096264\" (UID: \"b768b926-d65b-4ed7-afc0-bf5111096264\") " Nov 01 05:46:14.031752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.031725 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume" pod "b768b926-d65b-4ed7-afc0-bf5111096264" (UID: "b768b926-d65b-4ed7-afc0-bf5111096264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:14.032216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.032193 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b768b926-d65b-4ed7-afc0-bf5111096264-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:14.034407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.034384 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc" pod "b768b926-d65b-4ed7-afc0-bf5111096264" (UID: "b768b926-d65b-4ed7-afc0-bf5111096264"). InnerVolumeSpecName "kube-api-access-wk7nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:14.133562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.133529 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7nc\" (UniqueName: \"kubernetes.io/projected/b768b926-d65b-4ed7-afc0-bf5111096264-kube-api-access-wk7nc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:14.331696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.331663 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7"] Nov 01 05:46:14.332407 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:46:14.332378 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73344ae7_e8ce_456a_892c_5942fde5ff08.slice/crio-86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c WatchSource:0}: Error finding container 86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c: Status 404 returned error can't find the container with id 86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c Nov 01 05:46:14.671539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.671512 4425 generic.go:356] "Generic (PLEG): container finished" podID="b768b926-d65b-4ed7-afc0-bf5111096264" containerID="4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9" exitCode=0 Nov 01 05:46:14.672087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.671554 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" Nov 01 05:46:14.672087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.671590 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" event={"ID":"b768b926-d65b-4ed7-afc0-bf5111096264","Type":"ContainerDied","Data":"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9"} Nov 01 05:46:14.672087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.671647 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf" event={"ID":"b768b926-d65b-4ed7-afc0-bf5111096264","Type":"ContainerDied","Data":"950381cffc69562eb8a7dec53d5a5fdd42eeb9103a15caa7aed8faca7f488f64"} Nov 01 05:46:14.672087 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.671672 4425 scope.go:122] "RemoveContainer" containerID="4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9" Nov 01 05:46:14.673866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.673839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" event={"ID":"73344ae7-e8ce-456a-892c-5942fde5ff08","Type":"ContainerStarted","Data":"fd4bb27b1d98c81aa1a9aed43c6b873d83f0e31a956cb2f4e72de5d77b9d45f9"} Nov 01 05:46:14.673979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.673890 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" event={"ID":"73344ae7-e8ce-456a-892c-5942fde5ff08","Type":"ContainerStarted","Data":"86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c"} Nov 01 05:46:14.682908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.682887 4425 scope.go:122] "RemoveContainer" containerID="4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9" Nov 01 05:46:14.683189 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:14.683170 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9\": container with ID starting with 4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9 not found: ID does not exist" containerID="4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9" Nov 01 05:46:14.683308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.683196 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9"} err="failed to get container status \"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9\": rpc error: code = NotFound desc = could not find container \"4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9\": container with ID starting with 4fabe8da6cf1e0d65270f8a63a0552bfaa04e5e36686a2fb91cd9bf25e8580e9 not found: ID does not exist" Nov 01 05:46:14.830373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.830339 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:46:14.842853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.842800 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" podStartSLOduration=1.8427868219999999 podStartE2EDuration="1.842786822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:14.84232759 +0000 UTC m=+2589.982916532" watchObservedRunningTime="2026-11-01 05:46:14.842786822 +0000 UTC m=+2589.983375741" Nov 01 05:46:14.957495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.957413 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:46:14.960611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.960588 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8815d2-6b474787f-dcrwf"] Nov 01 05:46:14.973874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:14.973847 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b768b926-d65b-4ed7-afc0-bf5111096264" path="/var/lib/kubelet/pods/b768b926-d65b-4ed7-afc0-bf5111096264/volumes" Nov 01 05:46:15.706282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:15.705714 4425 generic.go:356] "Generic (PLEG): container finished" podID="73344ae7-e8ce-456a-892c-5942fde5ff08" containerID="fd4bb27b1d98c81aa1a9aed43c6b873d83f0e31a956cb2f4e72de5d77b9d45f9" exitCode=0 Nov 01 05:46:15.706282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:15.705944 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" event={"ID":"73344ae7-e8ce-456a-892c-5942fde5ff08","Type":"ContainerDied","Data":"fd4bb27b1d98c81aa1a9aed43c6b873d83f0e31a956cb2f4e72de5d77b9d45f9"} Nov 01 05:46:17.335350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.316221 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:17.525278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.524532 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\" (UniqueName: \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\") pod \"73344ae7-e8ce-456a-892c-5942fde5ff08\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " Nov 01 05:46:17.525278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.524737 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\") pod \"73344ae7-e8ce-456a-892c-5942fde5ff08\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " Nov 01 05:46:17.525278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.524808 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\") pod \"73344ae7-e8ce-456a-892c-5942fde5ff08\" (UID: \"73344ae7-e8ce-456a-892c-5942fde5ff08\") " Nov 01 05:46:17.535278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.525627 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca" pod "73344ae7-e8ce-456a-892c-5942fde5ff08" (UID: "73344ae7-e8ce-456a-892c-5942fde5ff08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:17.535278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.532745 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k" pod "73344ae7-e8ce-456a-892c-5942fde5ff08" (UID: "73344ae7-e8ce-456a-892c-5942fde5ff08"). InnerVolumeSpecName "kube-api-access-jkx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:17.536698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.536558 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca" pod "73344ae7-e8ce-456a-892c-5942fde5ff08" (UID: "73344ae7-e8ce-456a-892c-5942fde5ff08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:46:17.635617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.635518 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:17.635617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.635549 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73344ae7-e8ce-456a-892c-5942fde5ff08-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:17.635617 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.635565 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkx2k\" (UniqueName: \"kubernetes.io/projected/73344ae7-e8ce-456a-892c-5942fde5ff08-kube-api-access-jkx2k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:17.783898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.783824 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" event={"ID":"73344ae7-e8ce-456a-892c-5942fde5ff08","Type":"ContainerDied","Data":"86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c"} Nov 01 05:46:17.784206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.784155 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c54437bd6abcd2f9d6059a9a482b7cecab1a1a94fd88b7fd2e17b9ae4eb19c" Nov 01 05:46:17.784491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:17.784391 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7" Nov 01 05:46:19.831242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:19.831200 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" podUID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" containerName="payload-processing" containerID="cri-o://378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5" gracePeriod=25 Nov 01 05:46:20.144352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.144317 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:46:20.187489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.187454 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\") pod \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " Nov 01 05:46:20.187677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.187501 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\") pod \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " Nov 01 05:46:20.187677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.187627 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\") pod \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " Nov 01 05:46:20.187774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.187714 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\") pod \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\" (UID: \"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d\") " Nov 01 05:46:20.188018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.187994 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume" pod "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" (UID: "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:20.188116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.188091 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume" pod "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" (UID: "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:20.188230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.188215 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:20.188312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.188237 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:20.190047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.190022 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials" pod "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" (UID: "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:20.190142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.190118 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f" pod "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" (UID: "7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d"). InnerVolumeSpecName "kube-api-access-rnn7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:20.289415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.289380 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnn7f\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-kube-api-access-rnn7f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:20.289415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.289415 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:20.320620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.320586 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:20.321563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321537 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b768b926-d65b-4ed7-afc0-bf5111096264" containerName="payload-pre-processing" Nov 01 05:46:20.321563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321562 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b768b926-d65b-4ed7-afc0-bf5111096264" containerName="payload-pre-processing" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321592 4425 cpu_manager.go:414] "containerMap: removing container" podUID="73344ae7-e8ce-456a-892c-5942fde5ff08" containerName="revoke-keys" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321601 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73344ae7-e8ce-456a-892c-5942fde5ff08" containerName="revoke-keys" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321631 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" containerName="payload-processing" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321638 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" containerName="payload-processing" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321772 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="73344ae7-e8ce-456a-892c-5942fde5ff08" containerName="revoke-keys" Nov 01 05:46:20.321799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321789 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" containerName="payload-processing" Nov 01 05:46:20.322133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321806 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="990e9ab6-e3b5-4ead-a448-4fc047650186" containerName="istio-proxy" Nov 01 05:46:20.322133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.321817 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b768b926-d65b-4ed7-afc0-bf5111096264" containerName="payload-pre-processing" Nov 01 05:46:20.326263 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.326226 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.379001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.378924 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:20.381562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.381536 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d27da752f422\"" Nov 01 05:46:20.381711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.381662 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hjtww\"" Nov 01 05:46:20.390380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.390349 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.390536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.390389 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.390536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.390521 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjfd\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.390674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.390564 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.491711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.491660 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjfd\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.491711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.491718 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.491947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.491930 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.492039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.491969 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.492514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.492492 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.492674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.492653 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.494640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.494617 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.553981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.553950 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjfd\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\") pod \"payload-processing-external-model-59599f94ff-6qzvs\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.644311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.644207 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:20.803906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.803867 4425 generic.go:356] "Generic (PLEG): container finished" podID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" containerID="378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5" exitCode=0 Nov 01 05:46:20.804129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.803945 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" Nov 01 05:46:20.804129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.803949 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" event={"ID":"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d","Type":"ContainerDied","Data":"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5"} Nov 01 05:46:20.804129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.804010 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq" event={"ID":"7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d","Type":"ContainerDied","Data":"0495a344a4f526917b09fc5850a04b74033d899c1776346d44f556b4ab5ff6ec"} Nov 01 05:46:20.804129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.804034 4425 scope.go:122] "RemoveContainer" containerID="378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5" Nov 01 05:46:20.815150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.815124 4425 scope.go:122] "RemoveContainer" containerID="378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5" Nov 01 05:46:20.815440 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:20.815417 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5\": container with ID starting with 378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5 not found: ID does not exist" containerID="378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5" Nov 01 05:46:20.815498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.815449 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5"} err="failed to get container status \"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5\": rpc error: code = NotFound desc = could not find container \"378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5\": container with ID starting with 378535efb26f2de0a90151cbf1c4e819b542a6a1fe0dd316ede8053f41e2d9a5 not found: ID does not exist" Nov 01 05:46:20.974976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.974890 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:46:20.977714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:20.977696 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d67f47f4-bp4gq"] Nov 01 05:46:21.024575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:21.024542 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:21.062663 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:46:21.062636 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01239027_cfec_4a1a_b334_181cb5a00228.slice/crio-f2bb2d3b08757531eff6b55a5ed1a8bf2da6199d6cd49b9718f372210638bc36 WatchSource:0}: Error finding container f2bb2d3b08757531eff6b55a5ed1a8bf2da6199d6cd49b9718f372210638bc36: Status 404 returned error can't find the container with id f2bb2d3b08757531eff6b55a5ed1a8bf2da6199d6cd49b9718f372210638bc36 Nov 01 05:46:21.813447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:21.813406 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" event={"ID":"01239027-cfec-4a1a-b334-181cb5a00228","Type":"ContainerStarted","Data":"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b"} Nov 01 05:46:21.813643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:21.813471 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:21.813643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:21.813494 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" event={"ID":"01239027-cfec-4a1a-b334-181cb5a00228","Type":"ContainerStarted","Data":"f2bb2d3b08757531eff6b55a5ed1a8bf2da6199d6cd49b9718f372210638bc36"} Nov 01 05:46:21.933773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:21.933718 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" podStartSLOduration=1.9337036890000001 podStartE2EDuration="1.933703689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:21.931414269 +0000 UTC m=+2597.072003206" watchObservedRunningTime="2026-11-01 05:46:21.933703689 +0000 UTC m=+2597.074292608" Nov 01 05:46:22.921916 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:22.921871 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:22.921916 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:22.921902 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs: secret "agc-cred-d27da752f422" not found Nov 01 05:46:22.922434 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:22.921955 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials podName:01239027-cfec-4a1a-b334-181cb5a00228 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:23.421939105 +0000 UTC m=+2598.562528002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials") pod "payload-processing-external-model-59599f94ff-6qzvs" (UID: "01239027-cfec-4a1a-b334-181cb5a00228") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:22.974394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:22.974361 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d" path="/var/lib/kubelet/pods/7bbf1e48-4f16-4b31-bf16-d689e1f2fa4d/volumes" Nov 01 05:46:23.088587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:23.088543 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:23.429134 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:23.429089 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:23.429134 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:23.429126 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs: secret "agc-cred-d27da752f422" not found Nov 01 05:46:23.429476 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:23.429198 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials podName:01239027-cfec-4a1a-b334-181cb5a00228 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:24.429179888 +0000 UTC m=+2599.569768808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials") pod "payload-processing-external-model-59599f94ff-6qzvs" (UID: "01239027-cfec-4a1a-b334-181cb5a00228") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:24.442514 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:24.442480 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:24.442514 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:24.442510 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs: secret "agc-cred-d27da752f422" not found Nov 01 05:46:24.442958 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:24.442569 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials podName:01239027-cfec-4a1a-b334-181cb5a00228 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:26.442551462 +0000 UTC m=+2601.583140360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials") pod "payload-processing-external-model-59599f94ff-6qzvs" (UID: "01239027-cfec-4a1a-b334-181cb5a00228") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:25.746349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:25.746318 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:46:25.855060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:25.855028 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:46:26.467342 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:26.467307 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:26.467342 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:26.467332 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs: secret "agc-cred-d27da752f422" not found Nov 01 05:46:26.467603 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:26.467386 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials podName:01239027-cfec-4a1a-b334-181cb5a00228 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:30.467371655 +0000 UTC m=+2605.607960553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials") pod "payload-processing-external-model-59599f94ff-6qzvs" (UID: "01239027-cfec-4a1a-b334-181cb5a00228") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:26.948830 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:26.948796 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:46:28.828889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:28.828840 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" podUID="01239027-cfec-4a1a-b334-181cb5a00228" containerName="payload-processing" containerID="cri-o://6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b" gracePeriod=25 Nov 01 05:46:29.050900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.050859 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2"] Nov 01 05:46:29.100944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.100917 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f80f5ce-gkmf2"] Nov 01 05:46:29.105150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.105130 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:29.202554 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.202519 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\") pod \"01239027-cfec-4a1a-b334-181cb5a00228\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " Nov 01 05:46:29.202752 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.202594 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\") pod \"01239027-cfec-4a1a-b334-181cb5a00228\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " Nov 01 05:46:29.202819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.202807 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\") pod \"01239027-cfec-4a1a-b334-181cb5a00228\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " Nov 01 05:46:29.203065 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.203038 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\") pod \"01239027-cfec-4a1a-b334-181cb5a00228\" (UID: \"01239027-cfec-4a1a-b334-181cb5a00228\") " Nov 01 05:46:29.203326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.203295 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume" pod "01239027-cfec-4a1a-b334-181cb5a00228" (UID: "01239027-cfec-4a1a-b334-181cb5a00228"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:29.203326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.203312 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume" pod "01239027-cfec-4a1a-b334-181cb5a00228" (UID: "01239027-cfec-4a1a-b334-181cb5a00228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:29.203646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.203625 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:29.203715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.203649 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01239027-cfec-4a1a-b334-181cb5a00228-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:29.205453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.205430 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd" pod "01239027-cfec-4a1a-b334-181cb5a00228" (UID: "01239027-cfec-4a1a-b334-181cb5a00228"). InnerVolumeSpecName "kube-api-access-dsjfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:29.205545 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.205460 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials" pod "01239027-cfec-4a1a-b334-181cb5a00228" (UID: "01239027-cfec-4a1a-b334-181cb5a00228"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:29.305979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.305923 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:29.305979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.305957 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjfd\" (UniqueName: \"kubernetes.io/projected/01239027-cfec-4a1a-b334-181cb5a00228-kube-api-access-dsjfd\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:29.915699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.915660 4425 generic.go:356] "Generic (PLEG): container finished" podID="01239027-cfec-4a1a-b334-181cb5a00228" containerID="6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b" exitCode=0 Nov 01 05:46:29.916135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.915707 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" Nov 01 05:46:29.916135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.915732 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" event={"ID":"01239027-cfec-4a1a-b334-181cb5a00228","Type":"ContainerDied","Data":"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b"} Nov 01 05:46:29.916135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.915791 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs" event={"ID":"01239027-cfec-4a1a-b334-181cb5a00228","Type":"ContainerDied","Data":"f2bb2d3b08757531eff6b55a5ed1a8bf2da6199d6cd49b9718f372210638bc36"} Nov 01 05:46:29.916135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.915822 4425 scope.go:122] "RemoveContainer" containerID="6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b" Nov 01 05:46:29.926612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.926590 4425 scope.go:122] "RemoveContainer" containerID="6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b" Nov 01 05:46:29.926933 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:29.926907 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b\": container with ID starting with 6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b not found: ID does not exist" containerID="6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b" Nov 01 05:46:29.927043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:29.926943 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b"} err="failed to get container status \"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b\": rpc error: code = NotFound desc = could not find container \"6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b\": container with ID starting with 6b73b153d7cd11a4b2d49f5a64c2b3ea705a66bc0b0573e80bab23b9c89f279b not found: ID does not exist" Nov 01 05:46:30.084340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.084304 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:30.134853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.134817 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-6qzvs"] Nov 01 05:46:30.746670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.746629 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" podUID="b966c01f-27e5-4640-b691-a2a43e52a380" containerName="payload-processing" containerID="cri-o://238bbbc9e1cdfeb4b8a73efd15cd19062c8fb12dc34c261e98360a82a537ecc8" gracePeriod=25 Nov 01 05:46:30.855828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.855784 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" podUID="ff62ef87-25eb-419c-a142-ccae8827781b" containerName="payload-pre-processing" containerID="cri-o://19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8" gracePeriod=25 Nov 01 05:46:30.929264 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.929220 4425 generic.go:356] "Generic (PLEG): container finished" podID="b966c01f-27e5-4640-b691-a2a43e52a380" containerID="238bbbc9e1cdfeb4b8a73efd15cd19062c8fb12dc34c261e98360a82a537ecc8" exitCode=0 Nov 01 05:46:30.929703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.929288 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" event={"ID":"b966c01f-27e5-4640-b691-a2a43e52a380","Type":"ContainerDied","Data":"238bbbc9e1cdfeb4b8a73efd15cd19062c8fb12dc34c261e98360a82a537ecc8"} Nov 01 05:46:30.979526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.979493 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01239027-cfec-4a1a-b334-181cb5a00228" path="/var/lib/kubelet/pods/01239027-cfec-4a1a-b334-181cb5a00228/volumes" Nov 01 05:46:30.980129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:30.980111 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cd8998-782e-4942-8767-83e5c019c319" path="/var/lib/kubelet/pods/35cd8998-782e-4942-8767-83e5c019c319/volumes" Nov 01 05:46:31.020911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.020879 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:46:31.126913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.126863 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\") pod \"b966c01f-27e5-4640-b691-a2a43e52a380\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " Nov 01 05:46:31.127148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.126945 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\" (UniqueName: \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\") pod \"b966c01f-27e5-4640-b691-a2a43e52a380\" (UID: \"b966c01f-27e5-4640-b691-a2a43e52a380\") " Nov 01 05:46:31.127340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.127314 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume" pod "b966c01f-27e5-4640-b691-a2a43e52a380" (UID: "b966c01f-27e5-4640-b691-a2a43e52a380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:31.127480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.127464 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b966c01f-27e5-4640-b691-a2a43e52a380-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:31.129520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.129497 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn" pod "b966c01f-27e5-4640-b691-a2a43e52a380" (UID: "b966c01f-27e5-4640-b691-a2a43e52a380"). InnerVolumeSpecName "kube-api-access-9flzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:31.131349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.131327 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:46:31.228280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.228228 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\" (UniqueName: \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\") pod \"ff62ef87-25eb-419c-a142-ccae8827781b\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " Nov 01 05:46:31.228460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.228306 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\") pod \"ff62ef87-25eb-419c-a142-ccae8827781b\" (UID: \"ff62ef87-25eb-419c-a142-ccae8827781b\") " Nov 01 05:46:31.228660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.228644 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flzn\" (UniqueName: \"kubernetes.io/projected/b966c01f-27e5-4640-b691-a2a43e52a380-kube-api-access-9flzn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:31.228760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.228734 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume" pod "ff62ef87-25eb-419c-a142-ccae8827781b" (UID: "ff62ef87-25eb-419c-a142-ccae8827781b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:31.230661 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.230637 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js" pod "ff62ef87-25eb-419c-a142-ccae8827781b" (UID: "ff62ef87-25eb-419c-a142-ccae8827781b"). InnerVolumeSpecName "kube-api-access-hz2js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:31.330218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.330185 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2js\" (UniqueName: \"kubernetes.io/projected/ff62ef87-25eb-419c-a142-ccae8827781b-kube-api-access-hz2js\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:31.330218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.330213 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff62ef87-25eb-419c-a142-ccae8827781b-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:31.941404 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.941374 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" Nov 01 05:46:31.941867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.941416 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f" event={"ID":"b966c01f-27e5-4640-b691-a2a43e52a380","Type":"ContainerDied","Data":"3bfa1eefc7a4ca0e1e686f2713a89e94ae0c98942f8dd91934717574aab714b2"} Nov 01 05:46:31.941867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.941480 4425 scope.go:122] "RemoveContainer" containerID="238bbbc9e1cdfeb4b8a73efd15cd19062c8fb12dc34c261e98360a82a537ecc8" Nov 01 05:46:31.962037 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.962002 4425 generic.go:356] "Generic (PLEG): container finished" podID="ff62ef87-25eb-419c-a142-ccae8827781b" containerID="19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8" exitCode=0 Nov 01 05:46:31.962183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.962116 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" Nov 01 05:46:31.962321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.962297 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" event={"ID":"ff62ef87-25eb-419c-a142-ccae8827781b","Type":"ContainerDied","Data":"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8"} Nov 01 05:46:31.962428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.962407 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr" event={"ID":"ff62ef87-25eb-419c-a142-ccae8827781b","Type":"ContainerDied","Data":"699a347276769e560d977fa5a0145edce78a37df50eff38e49dcfbb08e233b61"} Nov 01 05:46:31.967116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.967090 4425 scope.go:122] "RemoveContainer" containerID="19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8" Nov 01 05:46:31.979021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.978877 4425 scope.go:122] "RemoveContainer" containerID="19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8" Nov 01 05:46:31.979272 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:31.979233 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8\": container with ID starting with 19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8 not found: ID does not exist" containerID="19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8" Nov 01 05:46:31.979359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:31.979288 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8"} err="failed to get container status \"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8\": rpc error: code = NotFound desc = could not find container \"19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8\": container with ID starting with 19200d36199c30ac08d65c609ed2c2a3a4895a7172e69d833b0a59d17fb214c8 not found: ID does not exist" Nov 01 05:46:32.109535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.109493 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:46:32.115655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.115617 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-31e728ff-76fbbf7c79-xwwmr"] Nov 01 05:46:32.225881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.225839 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:46:32.232458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.232424 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-31e728ff-b8958465f-cxf5f"] Nov 01 05:46:32.974380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.974344 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b966c01f-27e5-4640-b691-a2a43e52a380" path="/var/lib/kubelet/pods/b966c01f-27e5-4640-b691-a2a43e52a380/volumes" Nov 01 05:46:32.974811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:32.974793 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff62ef87-25eb-419c-a142-ccae8827781b" path="/var/lib/kubelet/pods/ff62ef87-25eb-419c-a142-ccae8827781b/volumes" Nov 01 05:46:35.517724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.517687 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z"] Nov 01 05:46:35.520078 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520046 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ff62ef87-25eb-419c-a142-ccae8827781b" containerName="payload-pre-processing" Nov 01 05:46:35.520295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520277 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff62ef87-25eb-419c-a142-ccae8827781b" containerName="payload-pre-processing" Nov 01 05:46:35.520436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520424 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b966c01f-27e5-4640-b691-a2a43e52a380" containerName="payload-processing" Nov 01 05:46:35.520523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520511 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b966c01f-27e5-4640-b691-a2a43e52a380" containerName="payload-processing" Nov 01 05:46:35.520637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520627 4425 cpu_manager.go:414] "containerMap: removing container" podUID="01239027-cfec-4a1a-b334-181cb5a00228" containerName="payload-processing" Nov 01 05:46:35.520719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520708 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01239027-cfec-4a1a-b334-181cb5a00228" containerName="payload-processing" Nov 01 05:46:35.520963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.520949 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="01239027-cfec-4a1a-b334-181cb5a00228" containerName="payload-processing" Nov 01 05:46:35.521097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.521084 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff62ef87-25eb-419c-a142-ccae8827781b" containerName="payload-pre-processing" Nov 01 05:46:35.521198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.521189 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b966c01f-27e5-4640-b691-a2a43e52a380" containerName="payload-processing" Nov 01 05:46:35.529822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.529792 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.619292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.614689 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 05:46:35.641435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.619675 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d27da752f422\"" Nov 01 05:46:35.641435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.620515 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ngk86\"" Nov 01 05:46:35.641435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.620590 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 05:46:35.641435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.620668 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 05:46:35.641435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.620727 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 05:46:35.650291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.642688 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78dz\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.652599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.652559 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.652760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.652631 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.652825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.652804 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.688429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.688393 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z"] Nov 01 05:46:35.790336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.790147 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.790336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.790222 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78dz\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.790601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.790334 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.790601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.790378 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.822894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.791430 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.822894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.792209 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:35.822894 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:35.821463 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:35.822894 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:35.821504 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z: secret "agc-cred-d27da752f422" not found Nov 01 05:46:35.822894 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:35.821587 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials podName:1385c137-6590-40f9-85d7-c20dd3ae1eb0 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:36.321549704 +0000 UTC m=+2611.462138620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials") pod "payload-processing-external-model-59599f94ff-dzx6z" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:35.937279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:35.937100 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78dz\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:36.030608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:36.030574 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z"] Nov 01 05:46:36.030927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:36.030906 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" podUID="1385c137-6590-40f9-85d7-c20dd3ae1eb0" Nov 01 05:46:36.325366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:36.325327 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:36.325569 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:36.325476 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:36.325569 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:36.325496 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z: secret "agc-cred-d27da752f422" not found Nov 01 05:46:36.325569 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:36.325567 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials podName:1385c137-6590-40f9-85d7-c20dd3ae1eb0 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:37.325551909 +0000 UTC m=+2612.466140823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials") pod "payload-processing-external-model-59599f94ff-dzx6z" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:37.036705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.036675 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:37.046064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.046037 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:37.136504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.136468 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\") pod \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " Nov 01 05:46:37.136683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.136590 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\") pod \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " Nov 01 05:46:37.136727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.136702 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\") pod \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " Nov 01 05:46:37.136974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.136891 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume" pod "1385c137-6590-40f9-85d7-c20dd3ae1eb0" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:37.137079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.137047 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume" pod "1385c137-6590-40f9-85d7-c20dd3ae1eb0" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:37.137226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.137213 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.137294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.137232 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1385c137-6590-40f9-85d7-c20dd3ae1eb0-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.139038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.139011 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz" pod "1385c137-6590-40f9-85d7-c20dd3ae1eb0" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0"). InnerVolumeSpecName "kube-api-access-j78dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:37.239230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.239176 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j78dz\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-kube-api-access-j78dz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:37.340804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:37.340760 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials\") pod \"payload-processing-external-model-59599f94ff-dzx6z\" (UID: \"1385c137-6590-40f9-85d7-c20dd3ae1eb0\") " pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:37.341021 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:37.340929 4425 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d27da752f422: secret "agc-cred-d27da752f422" not found Nov 01 05:46:37.341021 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:37.340952 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z: secret "agc-cred-d27da752f422" not found Nov 01 05:46:37.341021 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:37.341019 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials podName:1385c137-6590-40f9-85d7-c20dd3ae1eb0 nodeName:}" failed. No retries permitted until 2026-11-01 05:46:39.34100018 +0000 UTC m=+2614.481589082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials") pod "payload-processing-external-model-59599f94ff-dzx6z" (UID: "1385c137-6590-40f9-85d7-c20dd3ae1eb0") : secret "agc-cred-d27da752f422" not found Nov 01 05:46:38.043705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:38.043670 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z" Nov 01 05:46:38.276117 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:38.276082 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z"] Nov 01 05:46:38.282330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:38.282301 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59599f94ff-dzx6z"] Nov 01 05:46:38.353647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:38.353616 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1385c137-6590-40f9-85d7-c20dd3ae1eb0-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:38.973112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:38.973075 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1385c137-6590-40f9-85d7-c20dd3ae1eb0" path="/var/lib/kubelet/pods/1385c137-6590-40f9-85d7-c20dd3ae1eb0/volumes" Nov 01 05:46:41.137172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.137143 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z"] Nov 01 05:46:41.140229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.140192 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab085c4a-hfp5z"] Nov 01 05:46:41.213571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.213541 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:41.217831 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.217808 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.271766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.271728 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bcba75fe93c5\"" Nov 01 05:46:41.271944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.271884 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 05:46:41.272043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.272017 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t9dpj\"" Nov 01 05:46:41.272130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.272063 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 05:46:41.272130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.272108 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 05:46:41.272244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.272208 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 05:46:41.275950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.275923 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:41.296376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.296340 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.296543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.296501 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b44b\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.296674 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.296648 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.296748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.296710 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.397751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.397652 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b44b\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.397917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.397778 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.397917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.397827 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.398049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.397955 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.398691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.398603 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.398691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.398735 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.400741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.400685 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.459504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.459473 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b44b\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\") pod \"payload-processing-external-model-5565547b4b-2kj8t\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.531922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.531891 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:41.913077 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:46:41.913040 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1524de59_e615_455c_8791_72a300deb0d6.slice/crio-e7055805e6be261a37282771cfa0b62eb4db49278814ec022d07313d8a75707f WatchSource:0}: Error finding container e7055805e6be261a37282771cfa0b62eb4db49278814ec022d07313d8a75707f: Status 404 returned error can't find the container with id e7055805e6be261a37282771cfa0b62eb4db49278814ec022d07313d8a75707f Nov 01 05:46:41.913650 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:41.913575 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:42.075204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:42.075167 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" event={"ID":"1524de59-e615-455c-8791-72a300deb0d6","Type":"ContainerStarted","Data":"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60"} Nov 01 05:46:42.075412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:42.075228 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" event={"ID":"1524de59-e615-455c-8791-72a300deb0d6","Type":"ContainerStarted","Data":"e7055805e6be261a37282771cfa0b62eb4db49278814ec022d07313d8a75707f"} Nov 01 05:46:42.075412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:42.075287 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:42.195070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:42.194967 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" podStartSLOduration=1.194949179 podStartE2EDuration="1.194949179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:42.193243398 +0000 UTC m=+2617.333832317" watchObservedRunningTime="2026-11-01 05:46:42.194949179 +0000 UTC m=+2617.335538097" Nov 01 05:46:42.975166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:42.975122 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4f4df5-0fa1-4810-b6de-a59746093180" path="/var/lib/kubelet/pods/8e4f4df5-0fa1-4810-b6de-a59746093180/volumes" Nov 01 05:46:44.318195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:44.318157 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:48.134736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.134701 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:46:48.140204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.140179 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.196045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.196002 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.196525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.196500 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.196758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.196742 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.196950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.196937 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.197119 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.197105 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.197346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.197327 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.197696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.197676 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.200202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.199615 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.200202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.199687 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqv4\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.200202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.196431 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ad05c6-openshift-default-dockercfg-ptcxp\"" Nov 01 05:46:48.250443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.250412 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:46:48.301339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301295 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301361 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301432 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301476 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqv4\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301545 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301760 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301799 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301835 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301876 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.301969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.301940 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.302179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.302154 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.302246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.302200 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.302573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.302507 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.304332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.304303 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.304757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.304730 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.363603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.363572 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.363759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.363710 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqv4\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\") pod \"e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.454513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.454418 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:48.888696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.888667 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:46:48.889412 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:46:48.889387 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721ff6ee_8978_4adc_9bdf_fde443403d74.slice/crio-ea05451dec4b2f7e7ceed3a0e53935e29e49badd86da85796a1e17381f412acf WatchSource:0}: Error finding container ea05451dec4b2f7e7ceed3a0e53935e29e49badd86da85796a1e17381f412acf: Status 404 returned error can't find the container with id ea05451dec4b2f7e7ceed3a0e53935e29e49badd86da85796a1e17381f412acf Nov 01 05:46:48.892028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.891994 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:46:48.892111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.892066 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:46:48.892153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:48.892115 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:46:49.130640 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.130601 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerStarted","Data":"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff"} Nov 01 05:46:49.130853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.130699 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerStarted","Data":"ea05451dec4b2f7e7ceed3a0e53935e29e49badd86da85796a1e17381f412acf"} Nov 01 05:46:49.257275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.257134 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podStartSLOduration=1.257115821 podStartE2EDuration="1.257115821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:49.252287551 +0000 UTC m=+2624.392876466" watchObservedRunningTime="2026-11-01 05:46:49.257115821 +0000 UTC m=+2624.397704811" Nov 01 05:46:49.318578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.318532 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" podUID="1524de59-e615-455c-8791-72a300deb0d6" containerName="payload-processing" containerID="cri-o://6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60" gracePeriod=25 Nov 01 05:46:49.455036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.455000 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:49.461580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.461549 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:49.576581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.576549 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:49.719056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719014 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\") pod \"1524de59-e615-455c-8791-72a300deb0d6\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " Nov 01 05:46:49.719306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719203 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\") pod \"1524de59-e615-455c-8791-72a300deb0d6\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " Nov 01 05:46:49.719306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719276 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\") pod \"1524de59-e615-455c-8791-72a300deb0d6\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " Nov 01 05:46:49.719456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719317 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\") pod \"1524de59-e615-455c-8791-72a300deb0d6\" (UID: \"1524de59-e615-455c-8791-72a300deb0d6\") " Nov 01 05:46:49.719520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719471 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume" pod "1524de59-e615-455c-8791-72a300deb0d6" (UID: "1524de59-e615-455c-8791-72a300deb0d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:49.719804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719773 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume" pod "1524de59-e615-455c-8791-72a300deb0d6" (UID: "1524de59-e615-455c-8791-72a300deb0d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:46:49.719939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.719877 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:49.721626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.721604 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b" pod "1524de59-e615-455c-8791-72a300deb0d6" (UID: "1524de59-e615-455c-8791-72a300deb0d6"). InnerVolumeSpecName "kube-api-access-2b44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:49.721996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.721976 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials" pod "1524de59-e615-455c-8791-72a300deb0d6" (UID: "1524de59-e615-455c-8791-72a300deb0d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:46:49.821428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.821399 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b44b\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-kube-api-access-2b44b\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:49.821428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.821427 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1524de59-e615-455c-8791-72a300deb0d6-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:49.821634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:49.821438 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1524de59-e615-455c-8791-72a300deb0d6-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:46:50.138967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.138873 4425 generic.go:356] "Generic (PLEG): container finished" podID="1524de59-e615-455c-8791-72a300deb0d6" containerID="6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60" exitCode=0 Nov 01 05:46:50.138967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.138934 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" Nov 01 05:46:50.138967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.138951 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" event={"ID":"1524de59-e615-455c-8791-72a300deb0d6","Type":"ContainerDied","Data":"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60"} Nov 01 05:46:50.139325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.139002 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t" event={"ID":"1524de59-e615-455c-8791-72a300deb0d6","Type":"ContainerDied","Data":"e7055805e6be261a37282771cfa0b62eb4db49278814ec022d07313d8a75707f"} Nov 01 05:46:50.139325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.139026 4425 scope.go:122] "RemoveContainer" containerID="6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60" Nov 01 05:46:50.139476 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.139459 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:50.140937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.140911 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:46:50.149858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.149842 4425 scope.go:122] "RemoveContainer" containerID="6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60" Nov 01 05:46:50.150120 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:50.150098 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60\": container with ID starting with 6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60 not found: ID does not exist" containerID="6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60" Nov 01 05:46:50.150185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.150128 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60"} err="failed to get container status \"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60\": rpc error: code = NotFound desc = could not find container \"6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60\": container with ID starting with 6565bf54965f223c6338da4a6f18847ce12f63014b334565e58601a41a250b60 not found: ID does not exist" Nov 01 05:46:50.432462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.432381 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:50.439421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.439395 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5565547b4b-2kj8t"] Nov 01 05:46:50.973653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:50.973623 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1524de59-e615-455c-8791-72a300deb0d6" path="/var/lib/kubelet/pods/1524de59-e615-455c-8791-72a300deb0d6/volumes" Nov 01 05:46:56.048845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:56.048808 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw"] Nov 01 05:46:56.099830 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:56.099796 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa80308f-lwrbw"] Nov 01 05:46:56.948478 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:46:56.948440 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:46:56.973910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:56.973874 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf" path="/var/lib/kubelet/pods/6ef74f67-3f8a-4c6e-ab07-ff57135e8eaf/volumes" Nov 01 05:46:59.708781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.708743 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:46:59.709422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.709407 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1524de59-e615-455c-8791-72a300deb0d6" containerName="payload-processing" Nov 01 05:46:59.709474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.709423 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1524de59-e615-455c-8791-72a300deb0d6" containerName="payload-processing" Nov 01 05:46:59.709529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.709518 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1524de59-e615-455c-8791-72a300deb0d6" containerName="payload-processing" Nov 01 05:46:59.714790 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.714772 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:46:59.771154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.771128 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:46:59.839284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.824709 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ad05c6\"" Nov 01 05:46:59.839284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.824871 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ad05c6-dockercfg-k4w2t\"" Nov 01 05:46:59.853716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.853504 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:46:59.853716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.853577 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz777\" (UniqueName: \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:46:59.879490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.879456 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:46:59.883954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.883934 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:46:59.932389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.932362 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:46:59.954598 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.954566 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:46:59.954771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.954635 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz777\" (UniqueName: \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:46:59.955432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:46:59.955410 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:47:00.018477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.018399 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz777\" (UniqueName: \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\") pod \"payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:47:00.029326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.029288 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:47:00.056300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.056265 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bm8\" (UniqueName: \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.056653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.056609 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.157732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.157698 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.157913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.157798 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bm8\" (UniqueName: \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.158507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.158488 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.221333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.221302 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bm8\" (UniqueName: \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\") pod \"payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.408179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.408138 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:47:00.408334 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:00.408214 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7cf27c_0ea0_4b19_af83_5ea44a4a0ccc.slice/crio-245167a29d0340e343689485be3f26c93ec86a24b1c0c3a56af36b64aa66f2ef WatchSource:0}: Error finding container 245167a29d0340e343689485be3f26c93ec86a24b1c0c3a56af36b64aa66f2ef: Status 404 returned error can't find the container with id 245167a29d0340e343689485be3f26c93ec86a24b1c0c3a56af36b64aa66f2ef Nov 01 05:47:00.497176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.497150 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:00.877747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:00.877718 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:47:00.878125 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:00.877938 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfafc00_3cfa_44a4_8c78_528a18f3a4d9.slice/crio-8b6330378104fe585dadf5e29bf81a714f32fa2398ec5cb5b1e330f47c612038 WatchSource:0}: Error finding container 8b6330378104fe585dadf5e29bf81a714f32fa2398ec5cb5b1e330f47c612038: Status 404 returned error can't find the container with id 8b6330378104fe585dadf5e29bf81a714f32fa2398ec5cb5b1e330f47c612038 Nov 01 05:47:01.215320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.215208 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" event={"ID":"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9","Type":"ContainerStarted","Data":"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267"} Nov 01 05:47:01.215320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.215312 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" event={"ID":"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9","Type":"ContainerStarted","Data":"8b6330378104fe585dadf5e29bf81a714f32fa2398ec5cb5b1e330f47c612038"} Nov 01 05:47:01.215568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.215448 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:01.216856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.216832 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" event={"ID":"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc","Type":"ContainerStarted","Data":"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3"} Nov 01 05:47:01.216982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.216879 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:47:01.216982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.216897 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" event={"ID":"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc","Type":"ContainerStarted","Data":"245167a29d0340e343689485be3f26c93ec86a24b1c0c3a56af36b64aa66f2ef"} Nov 01 05:47:01.336389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.336331 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" podStartSLOduration=2.336317076 podStartE2EDuration="2.336317076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:01.334980899 +0000 UTC m=+2636.475569821" watchObservedRunningTime="2026-11-01 05:47:01.336317076 +0000 UTC m=+2636.476905994" Nov 01 05:47:01.457989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:01.457920 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" podStartSLOduration=2.4578941260000002 podStartE2EDuration="2.457894126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:01.453113814 +0000 UTC m=+2636.593702742" watchObservedRunningTime="2026-11-01 05:47:01.457894126 +0000 UTC m=+2636.598483046" Nov 01 05:47:07.049186 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:07.049152 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq"] Nov 01 05:47:07.099936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:07.099903 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea1b325-zbfsq"] Nov 01 05:47:07.119677 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:07.119650 4425 scope.go:122] "RemoveContainer" containerID="1464d75cefebff47f9fcec3eb873d42bea6f474ef24057cfe86f56b2a66d24c8" Nov 01 05:47:07.129983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:07.129960 4425 scope.go:122] "RemoveContainer" containerID="ef179d5ff308421e85c61b28a0f1883a1b4d3a854ceb55938932b90603acc443" Nov 01 05:47:08.974039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:08.974005 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d945ff35-d6e1-400c-bc12-89f7175e9abc" path="/var/lib/kubelet/pods/d945ff35-d6e1-400c-bc12-89f7175e9abc/volumes" Nov 01 05:47:12.227389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:12.227355 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:47:12.227824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:12.227442 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:47:24.863351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:24.863273 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7"] Nov 01 05:47:24.915732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:24.915699 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31e728ff-4skm7"] Nov 01 05:47:24.977778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:24.977733 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73344ae7-e8ce-456a-892c-5942fde5ff08" path="/var/lib/kubelet/pods/73344ae7-e8ce-456a-892c-5942fde5ff08/volumes" Nov 01 05:47:26.948387 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:47:26.948348 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:47:37.439905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.439863 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 05:47:37.444562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.444537 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.497608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.497579 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c73135\"" Nov 01 05:47:37.530810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.530753 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.531322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.531296 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mmh\" (UniqueName: \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.583881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.583766 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c73135-dockercfg-vwxmr\"" Nov 01 05:47:37.586355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.586328 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 05:47:37.608992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.608957 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 05:47:37.613291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.613269 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.634626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.634596 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.634789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.634714 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mmh\" (UniqueName: \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.635387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.635368 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.647620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.647591 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 05:47:37.696717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.696642 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mmh\" (UniqueName: \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\") pod \"payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.735762 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.735723 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpv4g\" (UniqueName: \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.735933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.735783 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.758541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.758512 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:37.839285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.837693 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpv4g\" (UniqueName: \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.839285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.837786 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.839285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.838727 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.900015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.899979 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpv4g\" (UniqueName: \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\") pod \"payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:37.928036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:37.928000 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:38.137940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.137914 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 05:47:38.138213 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:38.138175 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb139b5f9_155f_44ce_8488_02c0dfd3530f.slice/crio-f4438f56cc3bebfb065d8be7f8a7b922a3adf58b51be54a98ab0322d69e2f742 WatchSource:0}: Error finding container f4438f56cc3bebfb065d8be7f8a7b922a3adf58b51be54a98ab0322d69e2f742: Status 404 returned error can't find the container with id f4438f56cc3bebfb065d8be7f8a7b922a3adf58b51be54a98ab0322d69e2f742 Nov 01 05:47:38.305893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.305867 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 05:47:38.307234 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:38.307208 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd8ec32_8959_434b_bc5e_17911908e4c1.slice/crio-458c6f25624f9f8a2772212dfcad7023b1907eec75122f12b2caf6832ecc4e1c WatchSource:0}: Error finding container 458c6f25624f9f8a2772212dfcad7023b1907eec75122f12b2caf6832ecc4e1c: Status 404 returned error can't find the container with id 458c6f25624f9f8a2772212dfcad7023b1907eec75122f12b2caf6832ecc4e1c Nov 01 05:47:38.326799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.326771 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k"] Nov 01 05:47:38.333142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.333100 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.386471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.386450 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:47:38.386659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.386637 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:47:38.386761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.386688 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:47:38.386761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.386733 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:47:38.386950 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.386889 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k"] Nov 01 05:47:38.444940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.444908 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45k2\" (UniqueName: \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.445398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.445140 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.445398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.445198 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.515151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.515112 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" event={"ID":"b139b5f9-155f-44ce-8488-02c0dfd3530f","Type":"ContainerStarted","Data":"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e"} Nov 01 05:47:38.515400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.515176 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" event={"ID":"b139b5f9-155f-44ce-8488-02c0dfd3530f","Type":"ContainerStarted","Data":"f4438f56cc3bebfb065d8be7f8a7b922a3adf58b51be54a98ab0322d69e2f742"} Nov 01 05:47:38.515400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.515314 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:38.516936 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.516907 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" event={"ID":"0bd8ec32-8959-434b-bc5e-17911908e4c1","Type":"ContainerStarted","Data":"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9"} Nov 01 05:47:38.517059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.516950 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" event={"ID":"0bd8ec32-8959-434b-bc5e-17911908e4c1","Type":"ContainerStarted","Data":"458c6f25624f9f8a2772212dfcad7023b1907eec75122f12b2caf6832ecc4e1c"} Nov 01 05:47:38.517059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.516983 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:38.546781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.546754 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45k2\" (UniqueName: \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.546909 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.546897 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.546955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.546945 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.547531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.547515 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.549471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.549453 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.608612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.608588 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45k2\" (UniqueName: \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\") pod \"maas-api-revoke-keys-e2e-mt-25565930-ntj8k\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:38.635805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.635759 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" podStartSLOduration=1.635745879 podStartE2EDuration="1.635745879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:38.631817196 +0000 UTC m=+2673.772406140" watchObservedRunningTime="2026-11-01 05:47:38.635745879 +0000 UTC m=+2673.776334797" Nov 01 05:47:38.664990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:38.664957 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:39.040438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:39.040382 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" podStartSLOduration=2.040360336 podStartE2EDuration="2.040360336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:38.763220319 +0000 UTC m=+2673.903809238" watchObservedRunningTime="2026-11-01 05:47:39.040360336 +0000 UTC m=+2674.180949256" Nov 01 05:47:39.043477 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:39.043444 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4c64e5_c427_4ccf_9756_3785959a053d.slice/crio-252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480 WatchSource:0}: Error finding container 252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480: Status 404 returned error can't find the container with id 252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480 Nov 01 05:47:39.043618 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:39.043477 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k"] Nov 01 05:47:39.535720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:39.535683 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" event={"ID":"9b4c64e5-c427-4ccf-9756-3785959a053d","Type":"ContainerStarted","Data":"2f742bf3064f78d30fcd1a8467b92744f0cfec9f8a7256a88910d2e5c6dde4ee"} Nov 01 05:47:39.536125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:39.535762 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" event={"ID":"9b4c64e5-c427-4ccf-9756-3785959a053d","Type":"ContainerStarted","Data":"252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480"} Nov 01 05:47:39.655181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:39.655060 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" podStartSLOduration=1.655043687 podStartE2EDuration="1.655043687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:39.652780967 +0000 UTC m=+2674.793369885" watchObservedRunningTime="2026-11-01 05:47:39.655043687 +0000 UTC m=+2674.795632607" Nov 01 05:47:40.544804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:40.544768 4425 generic.go:356] "Generic (PLEG): container finished" podID="9b4c64e5-c427-4ccf-9756-3785959a053d" containerID="2f742bf3064f78d30fcd1a8467b92744f0cfec9f8a7256a88910d2e5c6dde4ee" exitCode=0 Nov 01 05:47:40.545222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:40.544854 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" event={"ID":"9b4c64e5-c427-4ccf-9756-3785959a053d","Type":"ContainerDied","Data":"2f742bf3064f78d30fcd1a8467b92744f0cfec9f8a7256a88910d2e5c6dde4ee"} Nov 01 05:47:41.708668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.708638 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:41.799981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.799950 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\") pod \"9b4c64e5-c427-4ccf-9756-3785959a053d\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " Nov 01 05:47:41.800154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.800059 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\") pod \"9b4c64e5-c427-4ccf-9756-3785959a053d\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " Nov 01 05:47:41.800154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.800093 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\" (UniqueName: \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\") pod \"9b4c64e5-c427-4ccf-9756-3785959a053d\" (UID: \"9b4c64e5-c427-4ccf-9756-3785959a053d\") " Nov 01 05:47:41.800376 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.800355 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca" pod "9b4c64e5-c427-4ccf-9756-3785959a053d" (UID: "9b4c64e5-c427-4ccf-9756-3785959a053d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:47:41.800818 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.800695 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:47:41.802488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.802461 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca" pod "9b4c64e5-c427-4ccf-9756-3785959a053d" (UID: "9b4c64e5-c427-4ccf-9756-3785959a053d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:47:41.802644 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.802602 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2" pod "9b4c64e5-c427-4ccf-9756-3785959a053d" (UID: "9b4c64e5-c427-4ccf-9756-3785959a053d"). InnerVolumeSpecName "kube-api-access-q45k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:47:41.902270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.902204 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4c64e5-c427-4ccf-9756-3785959a053d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:47:41.902270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:41.902245 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q45k2\" (UniqueName: \"kubernetes.io/projected/9b4c64e5-c427-4ccf-9756-3785959a053d-kube-api-access-q45k2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:47:42.560929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:42.560894 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" Nov 01 05:47:42.560929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:42.560913 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k" event={"ID":"9b4c64e5-c427-4ccf-9756-3785959a053d","Type":"ContainerDied","Data":"252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480"} Nov 01 05:47:42.561299 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:42.560973 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="252376d045b352a0afd3c7614edeb5bee0228154cfbf74951ae9ecd780ae7480" Nov 01 05:47:49.538809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:49.538777 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 05:47:49.539328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:49.538841 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 05:47:49.833224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:49.833192 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:47:49.945270 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:49.945226 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:47:54.834327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:54.834290 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" containerID="cri-o://6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91" gracePeriod=25 Nov 01 05:47:54.946413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:54.946352 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" containerID="cri-o://77740e5f9f6056aee84fe339b115cfa1e8d8860f535751e279825c6a8b24ade0" gracePeriod=25 Nov 01 05:47:56.952680 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:47:56.952638 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:47:59.101983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.101945 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:47:59.103533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.103504 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9b4c64e5-c427-4ccf-9756-3785959a053d" containerName="revoke-keys" Nov 01 05:47:59.103704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.103689 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4c64e5-c427-4ccf-9756-3785959a053d" containerName="revoke-keys" Nov 01 05:47:59.104021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.104002 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4c64e5-c427-4ccf-9756-3785959a053d" containerName="revoke-keys" Nov 01 05:47:59.116531 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.116506 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.169221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.164184 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:47:59.219472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.219436 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-0a08dbd11a06\"" Nov 01 05:47:59.228664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.228628 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff4l\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.228885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.228870 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.229080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.229059 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.229243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.229224 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.330492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.330451 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4l\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.330698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.330501 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.330698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.330686 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.330823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.330793 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.331760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.331723 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.331983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.331926 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.334821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.334801 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.419088 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.419011 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff4l\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.433269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.432998 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:47:59.944662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:47:59.944638 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:47:59.945369 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:47:59.945331 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7b6d43_3161_4677_a524_eb39493d4371.slice/crio-69d939124c670726beb611293d8a71683f4a19705a739c1e3fb678b0ac274042 WatchSource:0}: Error finding container 69d939124c670726beb611293d8a71683f4a19705a739c1e3fb678b0ac274042: Status 404 returned error can't find the container with id 69d939124c670726beb611293d8a71683f4a19705a739c1e3fb678b0ac274042 Nov 01 05:48:00.746438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:00.746398 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" event={"ID":"bd7b6d43-3161-4677-a524-eb39493d4371","Type":"ContainerStarted","Data":"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794"} Nov 01 05:48:00.746877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:00.746465 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" event={"ID":"bd7b6d43-3161-4677-a524-eb39493d4371","Type":"ContainerStarted","Data":"69d939124c670726beb611293d8a71683f4a19705a739c1e3fb678b0ac274042"} Nov 01 05:48:00.746877 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:00.746603 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:48:00.869064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:00.869007 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" podStartSLOduration=1.868995023 podStartE2EDuration="1.868995023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:48:00.864461849 +0000 UTC m=+2696.005050799" watchObservedRunningTime="2026-11-01 05:48:00.868995023 +0000 UTC m=+2696.009583941" Nov 01 05:48:02.938057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:02.938019 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:48:02.938057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:02.938031 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:48:06.883903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.883855 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:48:06.886878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.886854 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:48:06.934362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.934332 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:48:06.938499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.938480 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:06.994108 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.994075 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-21c241f95e7c\"" Nov 01 05:48:06.998638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:06.998609 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:48:07.037593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.037560 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhpw\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.037778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.037705 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.037778 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.037767 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.037985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.037963 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.139482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.139393 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.139673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.139497 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.139673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.139579 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhpw\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.139673 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.139659 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.140064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.140039 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.140291 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.140238 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.142435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.142404 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.200908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.200878 4425 scope.go:122] "RemoveContainer" containerID="65bb6ec57612583fd60a18e6c413ca93a213fbec939dec13928f51b70565b9f0" Nov 01 05:48:07.201792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.201765 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhpw\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.218932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.218903 4425 scope.go:122] "RemoveContainer" containerID="3796e320ce4ee0bca94a62685cfd5aa3add44f9e0cf93bba026a2720f2ac025d" Nov 01 05:48:07.228271 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.228237 4425 scope.go:122] "RemoveContainer" containerID="63de68ea7241550e97b429e83a4e00e6b9890dc5a372bf4aca6707f4b0fa17ec" Nov 01 05:48:07.237840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.237822 4425 scope.go:122] "RemoveContainer" containerID="fc2de4cd15dc97c13113b574a5a7bd62b1ecd23f039f0e43d8cd60003e93b164" Nov 01 05:48:07.251716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.251692 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.659160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.659130 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:48:07.659936 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:48:07.659910 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf006ad_a63a_4c53_8f9a_8f15a19cef58.slice/crio-8dbceea94b9323e6b36c233e0106baa22ef3755c2e75b01fc0e9d9420703cd18 WatchSource:0}: Error finding container 8dbceea94b9323e6b36c233e0106baa22ef3755c2e75b01fc0e9d9420703cd18: Status 404 returned error can't find the container with id 8dbceea94b9323e6b36c233e0106baa22ef3755c2e75b01fc0e9d9420703cd18 Nov 01 05:48:07.796738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.796699 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" event={"ID":"adf006ad-a63a-4c53-8f9a-8f15a19cef58","Type":"ContainerStarted","Data":"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2"} Nov 01 05:48:07.796940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.796760 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" event={"ID":"adf006ad-a63a-4c53-8f9a-8f15a19cef58","Type":"ContainerStarted","Data":"8dbceea94b9323e6b36c233e0106baa22ef3755c2e75b01fc0e9d9420703cd18"} Nov 01 05:48:07.796940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.796781 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:07.915190 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:07.915089 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" podStartSLOduration=1.9150751110000002 podStartE2EDuration="1.915075111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:48:07.914318006 +0000 UTC m=+2703.054906939" watchObservedRunningTime="2026-11-01 05:48:07.915075111 +0000 UTC m=+2703.055664063" Nov 01 05:48:11.885011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:11.884960 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" podUID="bd7b6d43-3161-4677-a524-eb39493d4371" containerName="payload-processing" containerID="cri-o://0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794" gracePeriod=25 Nov 01 05:48:12.186805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.182012 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:48:12.315733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.315695 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\") pod \"bd7b6d43-3161-4677-a524-eb39493d4371\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " Nov 01 05:48:12.315933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.315747 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\") pod \"bd7b6d43-3161-4677-a524-eb39493d4371\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " Nov 01 05:48:12.315933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.315880 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\") pod \"bd7b6d43-3161-4677-a524-eb39493d4371\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " Nov 01 05:48:12.316071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.315989 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\") pod \"bd7b6d43-3161-4677-a524-eb39493d4371\" (UID: \"bd7b6d43-3161-4677-a524-eb39493d4371\") " Nov 01 05:48:12.316131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.316067 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume" pod "bd7b6d43-3161-4677-a524-eb39493d4371" (UID: "bd7b6d43-3161-4677-a524-eb39493d4371"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:12.318157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.316513 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:12.318157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.316630 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume" pod "bd7b6d43-3161-4677-a524-eb39493d4371" (UID: "bd7b6d43-3161-4677-a524-eb39493d4371"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:12.321627 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.321592 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l" pod "bd7b6d43-3161-4677-a524-eb39493d4371" (UID: "bd7b6d43-3161-4677-a524-eb39493d4371"). InnerVolumeSpecName "kube-api-access-9ff4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:12.322129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.322099 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials" pod "bd7b6d43-3161-4677-a524-eb39493d4371" (UID: "bd7b6d43-3161-4677-a524-eb39493d4371"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:12.418039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.417960 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:12.418039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.417990 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7b6d43-3161-4677-a524-eb39493d4371-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:12.418039 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.418001 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ff4l\" (UniqueName: \"kubernetes.io/projected/bd7b6d43-3161-4677-a524-eb39493d4371-kube-api-access-9ff4l\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:12.842355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.842205 4425 generic.go:356] "Generic (PLEG): container finished" podID="bd7b6d43-3161-4677-a524-eb39493d4371" containerID="0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794" exitCode=0 Nov 01 05:48:12.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.842765 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" event={"ID":"bd7b6d43-3161-4677-a524-eb39493d4371","Type":"ContainerDied","Data":"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794"} Nov 01 05:48:12.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.842828 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" event={"ID":"bd7b6d43-3161-4677-a524-eb39493d4371","Type":"ContainerDied","Data":"69d939124c670726beb611293d8a71683f4a19705a739c1e3fb678b0ac274042"} Nov 01 05:48:12.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.842858 4425 scope.go:122] "RemoveContainer" containerID="0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794" Nov 01 05:48:12.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.843015 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h" Nov 01 05:48:12.884556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.884527 4425 scope.go:122] "RemoveContainer" containerID="0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794" Nov 01 05:48:12.885316 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:48:12.885281 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794\": container with ID starting with 0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794 not found: ID does not exist" containerID="0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794" Nov 01 05:48:12.885823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.885782 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794"} err="failed to get container status \"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794\": rpc error: code = NotFound desc = could not find container \"0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794\": container with ID starting with 0f8700ee836f3036601c251474f3c4a823949e88d4d3aef5a7064c78b804b794 not found: ID does not exist" Nov 01 05:48:12.940460 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.940420 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:48:12.940733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:12.940706 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:48:13.041283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:13.039054 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:48:13.057292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:13.056597 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdc2wd2h"] Nov 01 05:48:15.088127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:15.088089 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7b6d43-3161-4677-a524-eb39493d4371" path="/var/lib/kubelet/pods/bd7b6d43-3161-4677-a524-eb39493d4371/volumes" Nov 01 05:48:16.006437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:15.991914 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.010895 4425 generic.go:356] "Generic (PLEG): container finished" podID="5c457594-ca7c-4176-b445-3995642442aa" containerID="77740e5f9f6056aee84fe339b115cfa1e8d8860f535751e279825c6a8b24ade0" exitCode=0 Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.011120 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" event={"ID":"5c457594-ca7c-4176-b445-3995642442aa","Type":"ContainerDied","Data":"77740e5f9f6056aee84fe339b115cfa1e8d8860f535751e279825c6a8b24ade0"} Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.012739 4425 generic.go:356] "Generic (PLEG): container finished" podID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerID="6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91" exitCode=0 Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.012784 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" event={"ID":"fadeb7d4-ac45-4958-880a-56bc93e80682","Type":"ContainerDied","Data":"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91"} Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.012824 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" event={"ID":"fadeb7d4-ac45-4958-880a-56bc93e80682","Type":"ContainerDied","Data":"bfeb6ed4a5b8fcfe525af53341fa1c4dcc0f04d2c172a2a8ab07ee3d77274e7b"} Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.012853 4425 scope.go:122] "RemoveContainer" containerID="6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91" Nov 01 05:48:16.015415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.013156 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m" Nov 01 05:48:16.061605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.048592 4425 scope.go:122] "RemoveContainer" containerID="6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91" Nov 01 05:48:16.091096 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:48:16.089161 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91\": container with ID starting with 6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91 not found: ID does not exist" containerID="6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91" Nov 01 05:48:16.091096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.089209 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91"} err="failed to get container status \"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91\": rpc error: code = NotFound desc = could not find container \"6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91\": container with ID starting with 6d42ebf95df7630443e13cdf4432e581af709bb6edd0d0b154f8015688ff8c91 not found: ID does not exist" Nov 01 05:48:16.189612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.189575 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\" (UniqueName: \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\") pod \"fadeb7d4-ac45-4958-880a-56bc93e80682\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " Nov 01 05:48:16.189808 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.189645 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\") pod \"fadeb7d4-ac45-4958-880a-56bc93e80682\" (UID: \"fadeb7d4-ac45-4958-880a-56bc93e80682\") " Nov 01 05:48:16.192501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.192147 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:48:16.193665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.193638 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume" pod "fadeb7d4-ac45-4958-880a-56bc93e80682" (UID: "fadeb7d4-ac45-4958-880a-56bc93e80682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:16.194765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.194732 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq" pod "fadeb7d4-ac45-4958-880a-56bc93e80682" (UID: "fadeb7d4-ac45-4958-880a-56bc93e80682"). InnerVolumeSpecName "kube-api-access-fnfnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:16.291890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.291812 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnfnq\" (UniqueName: \"kubernetes.io/projected/fadeb7d4-ac45-4958-880a-56bc93e80682-kube-api-access-fnfnq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:16.291890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.291839 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadeb7d4-ac45-4958-880a-56bc93e80682-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:16.392629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.392590 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\" (UniqueName: \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\") pod \"5c457594-ca7c-4176-b445-3995642442aa\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " Nov 01 05:48:16.392835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.392724 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\") pod \"5c457594-ca7c-4176-b445-3995642442aa\" (UID: \"5c457594-ca7c-4176-b445-3995642442aa\") " Nov 01 05:48:16.393043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.393016 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume" pod "5c457594-ca7c-4176-b445-3995642442aa" (UID: "5c457594-ca7c-4176-b445-3995642442aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:16.393200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.393159 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c457594-ca7c-4176-b445-3995642442aa-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:16.395044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.395024 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9" pod "5c457594-ca7c-4176-b445-3995642442aa" (UID: "5c457594-ca7c-4176-b445-3995642442aa"). InnerVolumeSpecName "kube-api-access-ss5l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:16.440577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.440525 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:48:16.467624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.467585 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-lzl6m"] Nov 01 05:48:16.493875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.493845 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss5l9\" (UniqueName: \"kubernetes.io/projected/5c457594-ca7c-4176-b445-3995642442aa-kube-api-access-ss5l9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:16.975337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:16.975304 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" path="/var/lib/kubelet/pods/fadeb7d4-ac45-4958-880a-56bc93e80682/volumes" Nov 01 05:48:17.021325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:17.021291 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" event={"ID":"5c457594-ca7c-4176-b445-3995642442aa","Type":"ContainerDied","Data":"a808a4f8a2425ad1532389d72a213724b1fdd51c4c1fc6f964b51111e740d083"} Nov 01 05:48:17.021490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:17.021354 4425 scope.go:122] "RemoveContainer" containerID="77740e5f9f6056aee84fe339b115cfa1e8d8860f535751e279825c6a8b24ade0" Nov 01 05:48:17.021490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:17.021303 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j" Nov 01 05:48:17.193058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:17.192971 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:48:17.196886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:17.196860 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-gqg7j"] Nov 01 05:48:18.807046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:18.807021 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:48:18.960623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:18.960589 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:48:18.977420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:18.977387 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c457594-ca7c-4176-b445-3995642442aa" path="/var/lib/kubelet/pods/5c457594-ca7c-4176-b445-3995642442aa/volumes" Nov 01 05:48:21.181439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:21.181404 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 05:48:21.181916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:21.181465 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 05:48:23.065742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.065708 4425 generic.go:356] "Generic (PLEG): container finished" podID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerID="ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff" exitCode=0 Nov 01 05:48:23.066187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.065779 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerDied","Data":"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff"} Nov 01 05:48:23.066187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.066174 4425 scope.go:122] "RemoveContainer" containerID="ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff" Nov 01 05:48:23.067688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.067658 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:48:23.067739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.067730 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:48:23.067774 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.067763 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:48:23.966410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:23.966358 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" podUID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" containerName="payload-processing" containerID="cri-o://6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2" gracePeriod=25 Nov 01 05:48:24.075639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.075606 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerStarted","Data":"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7"} Nov 01 05:48:24.281393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.281365 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:48:24.394183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394154 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\") pod \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " Nov 01 05:48:24.394448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394295 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\") pod \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " Nov 01 05:48:24.394448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394395 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\") pod \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " Nov 01 05:48:24.394448 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394437 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\") pod \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\" (UID: \"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e\") " Nov 01 05:48:24.394715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394636 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume" pod "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" (UID: "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:24.394773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394704 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume" pod "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" (UID: "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:24.394854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394838 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:24.394891 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.394857 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:24.396694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.396664 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials" pod "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" (UID: "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:24.396694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.396673 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z" pod "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" (UID: "51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e"). InnerVolumeSpecName "kube-api-access-tk26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:24.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.456692 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:48:24.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.456830 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 05:48:24.465279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.456892 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 05:48:24.496633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.496558 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk26z\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-kube-api-access-tk26z\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:24.496633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:24.496586 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:25.084016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.083976 4425 generic.go:356] "Generic (PLEG): container finished" podID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" containerID="6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2" exitCode=0 Nov 01 05:48:25.084511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.084028 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" Nov 01 05:48:25.084511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.084062 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" event={"ID":"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e","Type":"ContainerDied","Data":"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2"} Nov 01 05:48:25.084511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.084122 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn" event={"ID":"51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e","Type":"ContainerDied","Data":"d51a3b36bd0ff0605f37076332e248186bff8a39ecd992802aa55ab8bc97697a"} Nov 01 05:48:25.084511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.084148 4425 scope.go:122] "RemoveContainer" containerID="6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2" Nov 01 05:48:25.097507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.097294 4425 scope.go:122] "RemoveContainer" containerID="6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2" Nov 01 05:48:25.097627 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:48:25.097606 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2\": container with ID starting with 6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2 not found: ID does not exist" containerID="6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2" Nov 01 05:48:25.097672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.097637 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2"} err="failed to get container status \"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2\": rpc error: code = NotFound desc = could not find container \"6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2\": container with ID starting with 6f5897d06d4ac75ce4e06044ecd554811ddc6dd2943387d40dc311d39278bab2 not found: ID does not exist" Nov 01 05:48:25.268367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.268281 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:48:25.303807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.303773 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f98fvvn"] Nov 01 05:48:25.458692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.458653 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 05:48:25.458995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:25.458962 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 05:48:26.462727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:26.462685 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 05:48:26.463918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:26.463884 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 05:48:26.975909 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:48:26.975873 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:48:26.976296 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:26.976273 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" path="/var/lib/kubelet/pods/51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e/volumes" Nov 01 05:48:27.463394 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:27.463357 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 05:48:27.463995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:27.463963 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 05:48:28.466438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:28.466396 4425 patch_prober.go:29] interesting pod/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 05:48:28.466964 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:28.466457 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 05:48:29.459525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:29.459497 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:48:29.459805 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:29.459789 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:48:29.460601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:29.460580 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:48:44.073597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.073567 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx"] Nov 01 05:48:44.074165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074152 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" Nov 01 05:48:44.074217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074166 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" Nov 01 05:48:44.074217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074181 4425 cpu_manager.go:414] "containerMap: removing container" podUID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" containerName="payload-processing" Nov 01 05:48:44.074217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074186 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" containerName="payload-processing" Nov 01 05:48:44.074217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074209 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bd7b6d43-3161-4677-a524-eb39493d4371" containerName="payload-processing" Nov 01 05:48:44.074217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074214 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7b6d43-3161-4677-a524-eb39493d4371" containerName="payload-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074243 4425 cpu_manager.go:414] "containerMap: removing container" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074265 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074351 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7b6d43-3161-4677-a524-eb39493d4371" containerName="payload-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074360 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadeb7d4-ac45-4958-880a-56bc93e80682" containerName="payload-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074370 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c457594-ca7c-4176-b445-3995642442aa" containerName="payload-pre-processing" Nov 01 05:48:44.074415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.074376 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df0dd7-b2fb-4b59-a0cd-49e7a1c47a3e" containerName="payload-processing" Nov 01 05:48:44.078183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.078161 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.134651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.134619 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx"] Nov 01 05:48:44.161220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.161169 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.161443 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.161403 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.161512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.161442 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42m8m\" (UniqueName: \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.181363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.181327 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:48:44.181548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.181363 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:48:44.181548 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.181452 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:48:44.182967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.182952 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:48:44.262960 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.262925 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.263168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.262979 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42m8m\" (UniqueName: \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.263168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.263145 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.263741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.263720 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.265747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.265726 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.325237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.325161 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42m8m\" (UniqueName: \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\") pod \"maas-api-revoke-sub-5cf1be13-2xpbx\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.392539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.392505 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:44.769749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:44.769710 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx"] Nov 01 05:48:44.770336 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:48:44.770305 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ced1182_cedf_46f6_bdce_f38f5ab0fecd.slice/crio-656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad WatchSource:0}: Error finding container 656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad: Status 404 returned error can't find the container with id 656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad Nov 01 05:48:45.051994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.051923 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr"] Nov 01 05:48:45.054876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.054854 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9578fbd-k6nmr"] Nov 01 05:48:45.231942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.231902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" event={"ID":"6ced1182-cedf-46f6-bdce-f38f5ab0fecd","Type":"ContainerStarted","Data":"fba5d470d030798107a0e1d5723e2096324734c10b86e039c4f93ddaf9c3dbac"} Nov 01 05:48:45.232539 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.231964 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" event={"ID":"6ced1182-cedf-46f6-bdce-f38f5ab0fecd","Type":"ContainerStarted","Data":"656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad"} Nov 01 05:48:45.349757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.349704 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" podStartSLOduration=1.349689297 podStartE2EDuration="1.349689297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:48:45.348374834 +0000 UTC m=+2740.488963774" watchObservedRunningTime="2026-11-01 05:48:45.349689297 +0000 UTC m=+2740.490278215" Nov 01 05:48:45.475311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.475282 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:48:45.479610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.479589 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.533165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533136 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d7505c667e98\"" Nov 01 05:48:45.533165 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533143 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-s59hh\"" Nov 01 05:48:45.533533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533502 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 05:48:45.533533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533533 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 05:48:45.533533 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533550 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 05:48:45.533785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.533606 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 05:48:45.537484 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.537461 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:48:45.578897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.578854 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.578897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.578902 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbxm\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.579125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.579102 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.579176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.579151 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.680315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.680198 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.680492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.680339 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.680492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.680374 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbxm\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.680610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.680573 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.681233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.681115 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.681233 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.681167 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.682954 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.682933 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.742483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.742460 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbxm\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\") pod \"payload-processing-external-model-ff8dd698-cvrtg\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:45.794305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:45.794247 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:46.177164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:46.177136 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:48:46.177530 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:48:46.177506 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbe2934_2d38_4a80_b564_d1334ac50f26.slice/crio-3da955ba70f6d14834141282c9015775af3f5cac794e28f6866f7a8995e20e4b WatchSource:0}: Error finding container 3da955ba70f6d14834141282c9015775af3f5cac794e28f6866f7a8995e20e4b: Status 404 returned error can't find the container with id 3da955ba70f6d14834141282c9015775af3f5cac794e28f6866f7a8995e20e4b Nov 01 05:48:46.240536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:46.240503 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" event={"ID":"abbe2934-2d38-4a80-b564-d1334ac50f26","Type":"ContainerStarted","Data":"3da955ba70f6d14834141282c9015775af3f5cac794e28f6866f7a8995e20e4b"} Nov 01 05:48:46.242398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:46.242374 4425 generic.go:356] "Generic (PLEG): container finished" podID="6ced1182-cedf-46f6-bdce-f38f5ab0fecd" containerID="fba5d470d030798107a0e1d5723e2096324734c10b86e039c4f93ddaf9c3dbac" exitCode=0 Nov 01 05:48:46.242517 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:46.242430 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" event={"ID":"6ced1182-cedf-46f6-bdce-f38f5ab0fecd","Type":"ContainerDied","Data":"fba5d470d030798107a0e1d5723e2096324734c10b86e039c4f93ddaf9c3dbac"} Nov 01 05:48:46.974116 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:46.974080 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721b275d-d9b9-44a1-83b7-3d2b33f22dbe" path="/var/lib/kubelet/pods/721b275d-d9b9-44a1-83b7-3d2b33f22dbe/volumes" Nov 01 05:48:47.251000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.250902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" event={"ID":"abbe2934-2d38-4a80-b564-d1334ac50f26","Type":"ContainerStarted","Data":"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa"} Nov 01 05:48:47.251000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.250993 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:48:47.374708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.374650 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" podStartSLOduration=2.374634781 podStartE2EDuration="2.374634781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:48:47.372477739 +0000 UTC m=+2742.513066677" watchObservedRunningTime="2026-11-01 05:48:47.374634781 +0000 UTC m=+2742.515223698" Nov 01 05:48:47.414537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.414512 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:47.505384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.505300 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\") pod \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " Nov 01 05:48:47.505576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.505519 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\") pod \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " Nov 01 05:48:47.505576 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.505555 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\" (UniqueName: \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\") pod \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\" (UID: \"6ced1182-cedf-46f6-bdce-f38f5ab0fecd\") " Nov 01 05:48:47.505894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.505863 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca" pod "6ced1182-cedf-46f6-bdce-f38f5ab0fecd" (UID: "6ced1182-cedf-46f6-bdce-f38f5ab0fecd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:48:47.506154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.506135 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:47.507784 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.507759 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m" pod "6ced1182-cedf-46f6-bdce-f38f5ab0fecd" (UID: "6ced1182-cedf-46f6-bdce-f38f5ab0fecd"). InnerVolumeSpecName "kube-api-access-42m8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:48:47.507896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.507852 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca" pod "6ced1182-cedf-46f6-bdce-f38f5ab0fecd" (UID: "6ced1182-cedf-46f6-bdce-f38f5ab0fecd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:48:47.607847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.607810 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42m8m\" (UniqueName: \"kubernetes.io/projected/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-kube-api-access-42m8m\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:47.607847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:47.607845 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ced1182-cedf-46f6-bdce-f38f5ab0fecd-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:48:48.259447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:48.259417 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" Nov 01 05:48:48.259844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:48.259446 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx" event={"ID":"6ced1182-cedf-46f6-bdce-f38f5ab0fecd","Type":"ContainerDied","Data":"656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad"} Nov 01 05:48:48.259844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:48.259499 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="656a031bbcf94ed2955937f909ec716d741b76c9f1165385847947395ae9b5ad" Nov 01 05:48:56.946972 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:48:56.946934 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:48:57.132248 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:57.132211 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7"] Nov 01 05:48:57.146276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:57.146232 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2285bbb6-rgzg7"] Nov 01 05:48:58.262734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:58.262683 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:48:58.974038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:48:58.974006 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d1711d-a5ed-4208-a03e-575df3f53736" path="/var/lib/kubelet/pods/64d1711d-a5ed-4208-a03e-575df3f53736/volumes" Nov 01 05:49:08.260813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:08.260778 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:49:09.052011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:09.051976 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6"] Nov 01 05:49:09.102730 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:09.102700 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93010f46-5p6k6"] Nov 01 05:49:10.974976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:10.974940 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66751a77-c23f-40bc-bf6f-fb04056f5d28" path="/var/lib/kubelet/pods/66751a77-c23f-40bc-bf6f-fb04056f5d28/volumes" Nov 01 05:49:13.607560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.607523 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:49:13.609906 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.609855 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:49:13.638519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.638482 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:49:13.639114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.639099 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6ced1182-cedf-46f6-bdce-f38f5ab0fecd" containerName="revoke-keys" Nov 01 05:49:13.639164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.639116 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ced1182-cedf-46f6-bdce-f38f5ab0fecd" containerName="revoke-keys" Nov 01 05:49:13.639218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.639209 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ced1182-cedf-46f6-bdce-f38f5ab0fecd" containerName="revoke-keys" Nov 01 05:49:13.644311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.644292 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.697980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.697950 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d34a655eba7f\"" Nov 01 05:49:13.697980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.697950 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-30047653ba10\"" Nov 01 05:49:13.714692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.714660 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.714851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.714725 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.718490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.718453 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.718716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.718695 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntl5d\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.720651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.720621 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:49:13.819646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.819611 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntl5d\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.819828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.819670 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.819828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.819715 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.819828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.819744 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.820437 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.820413 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.820572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.820556 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.822665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.822638 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.881822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.881745 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntl5d\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\") pod \"payload-processing-external-model-6c9f596676-2bds7\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:13.958958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:13.958913 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:14.339691 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:14.339655 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:49:14.340382 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:49:14.340349 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8188d9_7fa6_4c6f_959c_e6fe7e0fe91d.slice/crio-11449111cf655b3508cf03d23c6e1393cf07d2f481ff819e2a1cf67d8945e4e0 WatchSource:0}: Error finding container 11449111cf655b3508cf03d23c6e1393cf07d2f481ff819e2a1cf67d8945e4e0: Status 404 returned error can't find the container with id 11449111cf655b3508cf03d23c6e1393cf07d2f481ff819e2a1cf67d8945e4e0 Nov 01 05:49:14.471908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:14.471874 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" event={"ID":"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d","Type":"ContainerStarted","Data":"c1086ca006b0099e1785d2ff0ec04e549ce3ab1fb02876e2a7ab0b2c873b6128"} Nov 01 05:49:14.472097 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:14.471939 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" event={"ID":"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d","Type":"ContainerStarted","Data":"11449111cf655b3508cf03d23c6e1393cf07d2f481ff819e2a1cf67d8945e4e0"} Nov 01 05:49:14.472160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:14.472099 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:14.591599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:14.591485 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" podStartSLOduration=1.591470607 podStartE2EDuration="1.591470607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:49:14.590394497 +0000 UTC m=+2769.730983428" watchObservedRunningTime="2026-11-01 05:49:14.591470607 +0000 UTC m=+2769.732059557" Nov 01 05:49:18.612278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:18.609350 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" containerID="cri-o://8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa" gracePeriod=25 Nov 01 05:49:19.208072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.207676 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:49:19.296229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.296145 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\") pod \"abbe2934-2d38-4a80-b564-d1334ac50f26\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " Nov 01 05:49:19.296229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.296186 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\") pod \"abbe2934-2d38-4a80-b564-d1334ac50f26\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " Nov 01 05:49:19.296510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.296379 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\") pod \"abbe2934-2d38-4a80-b564-d1334ac50f26\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " Nov 01 05:49:19.296577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.296539 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\") pod \"abbe2934-2d38-4a80-b564-d1334ac50f26\" (UID: \"abbe2934-2d38-4a80-b564-d1334ac50f26\") " Nov 01 05:49:19.296692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.296663 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume" pod "abbe2934-2d38-4a80-b564-d1334ac50f26" (UID: "abbe2934-2d38-4a80-b564-d1334ac50f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:49:19.297164 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.297112 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume" pod "abbe2934-2d38-4a80-b564-d1334ac50f26" (UID: "abbe2934-2d38-4a80-b564-d1334ac50f26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:49:19.298175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.298150 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:49:19.298311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.298182 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbe2934-2d38-4a80-b564-d1334ac50f26-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:49:19.301144 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.301097 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials" pod "abbe2934-2d38-4a80-b564-d1334ac50f26" (UID: "abbe2934-2d38-4a80-b564-d1334ac50f26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:49:19.301309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.301289 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm" pod "abbe2934-2d38-4a80-b564-d1334ac50f26" (UID: "abbe2934-2d38-4a80-b564-d1334ac50f26"). InnerVolumeSpecName "kube-api-access-5pbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:49:19.399901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.399862 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbxm\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-kube-api-access-5pbxm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:49:19.399901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.399892 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abbe2934-2d38-4a80-b564-d1334ac50f26-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:49:19.553510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.553415 4425 generic.go:356] "Generic (PLEG): container finished" podID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerID="8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa" exitCode=0 Nov 01 05:49:19.553510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.553468 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" Nov 01 05:49:19.553725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.553507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" event={"ID":"abbe2934-2d38-4a80-b564-d1334ac50f26","Type":"ContainerDied","Data":"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa"} Nov 01 05:49:19.553725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.553567 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg" event={"ID":"abbe2934-2d38-4a80-b564-d1334ac50f26","Type":"ContainerDied","Data":"3da955ba70f6d14834141282c9015775af3f5cac794e28f6866f7a8995e20e4b"} Nov 01 05:49:19.553725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.553596 4425 scope.go:122] "RemoveContainer" containerID="8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa" Nov 01 05:49:19.564768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.564749 4425 scope.go:122] "RemoveContainer" containerID="8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa" Nov 01 05:49:19.565113 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:49:19.565073 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa\": container with ID starting with 8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa not found: ID does not exist" containerID="8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa" Nov 01 05:49:19.565194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.565118 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa"} err="failed to get container status \"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa\": rpc error: code = NotFound desc = could not find container \"8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa\": container with ID starting with 8ef4312f6aa031565759bc60a62e9a2458899282f677b526688d9d96aab0d1fa not found: ID does not exist" Nov 01 05:49:19.724558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.724522 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:49:19.773772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:19.773741 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-cvrtg"] Nov 01 05:49:20.973700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:20.973669 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" path="/var/lib/kubelet/pods/abbe2934-2d38-4a80-b564-d1334ac50f26/volumes" Nov 01 05:49:24.048862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:24.048828 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v"] Nov 01 05:49:24.099492 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:24.099459 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-707d87a7-bww8v"] Nov 01 05:49:24.973819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:24.973776 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1c71cb-9479-42c9-98e7-b476be5a2035" path="/var/lib/kubelet/pods/3f1c71cb-9479-42c9-98e7-b476be5a2035/volumes" Nov 01 05:49:25.482619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:25.482592 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:49:26.948846 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:49:26.948803 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:49:37.055145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:37.055105 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp"] Nov 01 05:49:37.105789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:37.105756 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33eb21d5-7p2pp"] Nov 01 05:49:38.973642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:38.973605 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5a6546-3811-45af-8afc-1f4cfbd9ff4d" path="/var/lib/kubelet/pods/1c5a6546-3811-45af-8afc-1f4cfbd9ff4d/volumes" Nov 01 05:49:43.488275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.483883 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:49:43.488275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.484858 4425 cpu_manager.go:414] "containerMap: removing container" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" Nov 01 05:49:43.488275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.484880 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" Nov 01 05:49:43.488275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.485080 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbe2934-2d38-4a80-b564-d1334ac50f26" containerName="payload-processing" Nov 01 05:49:43.508286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.507446 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.632337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.632301 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.632657 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.632631 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.632838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.632821 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.633380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.632462 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:49:43.634389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.632997 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.634510 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.634424 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.634624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.634550 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.634624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.634580 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.634789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.634732 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.634789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.634775 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwwl\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.702856 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.702795 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7a360a-openshift-default-dockercfg-ggnz9\"" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762110 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762163 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwwl\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762197 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762301 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762370 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762449 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762474 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762601 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.762632 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.763843 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.763157 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.839224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.837712 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.853407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.852963 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwwl\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.862469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.862399 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.862651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.862574 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.862850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.862828 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.863095 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.863052 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.876392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.876333 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:43.885596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:43.885527 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:44.139607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.139567 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:44.578620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.578597 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:49:44.579836 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:49:44.579808 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1ea8b8_3f8c_4b9a_993c_b73797171178.slice/crio-af7bbf056f965d222ddbcc0345b3aa51203690a21055069545a509f24c348292 WatchSource:0}: Error finding container af7bbf056f965d222ddbcc0345b3aa51203690a21055069545a509f24c348292: Status 404 returned error can't find the container with id af7bbf056f965d222ddbcc0345b3aa51203690a21055069545a509f24c348292 Nov 01 05:49:44.582972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.582937 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:49:44.583054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.583007 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:49:44.583054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.583038 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:49:44.849586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.849487 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerStarted","Data":"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d"} Nov 01 05:49:44.849586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:44.849556 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerStarted","Data":"af7bbf056f965d222ddbcc0345b3aa51203690a21055069545a509f24c348292"} Nov 01 05:49:45.140500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:45.140417 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:45.146149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:45.146125 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:45.270053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:45.269997 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podStartSLOduration=2.269983444 podStartE2EDuration="2.269983444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:49:44.974386405 +0000 UTC m=+2800.114975338" watchObservedRunningTime="2026-11-01 05:49:45.269983444 +0000 UTC m=+2800.410572366" Nov 01 05:49:45.856038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:45.856007 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:45.856911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:45.856892 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:49:50.048516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.048481 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp"] Nov 01 05:49:50.098940 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.098904 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c5d33c-mp6kp"] Nov 01 05:49:50.146607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.146579 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k"] Nov 01 05:49:50.149433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.149397 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25565930-ntj8k"] Nov 01 05:49:50.974427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.974396 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b6083a-a488-4112-b9b7-37741243d7f6" path="/var/lib/kubelet/pods/15b6083a-a488-4112-b9b7-37741243d7f6/volumes" Nov 01 05:49:50.974834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:49:50.974817 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4c64e5-c427-4ccf-9756-3785959a053d" path="/var/lib/kubelet/pods/9b4c64e5-c427-4ccf-9756-3785959a053d/volumes" Nov 01 05:49:56.948556 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:49:56.948510 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:50:07.329047 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:07.329013 4425 scope.go:122] "RemoveContainer" containerID="143bbe3c740444ce1b7b199941dc04516dfa6e112cd6445d24bd0f989a54d970" Nov 01 05:50:07.339343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:07.339323 4425 scope.go:122] "RemoveContainer" containerID="57aee33a83f15110f3c51f0a7d4280c52862d32e9bfca5e60193b39f25dcc80f" Nov 01 05:50:07.348773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:07.348749 4425 scope.go:122] "RemoveContainer" containerID="6e604ffb0beeeabcd11a1b2cfed2b122e27c5658d98440b7b24e0fae25baf1f1" Nov 01 05:50:07.358403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:07.358382 4425 scope.go:122] "RemoveContainer" containerID="5b91afd8ae702b3ae558d5eea896e4794650bf0cad300ca9e3f90455115eb471" Nov 01 05:50:26.948032 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:50:26.947947 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:50:42.871268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:42.871221 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:50:42.980557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:42.980527 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:50:47.872355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:47.872281 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" podUID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" containerName="payload-processing" containerID="cri-o://f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267" gracePeriod=25 Nov 01 05:50:47.981345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:47.981303 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" podUID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" containerName="payload-pre-processing" containerID="cri-o://fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3" gracePeriod=25 Nov 01 05:50:48.153888 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.153861 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:50:48.262059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.262029 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\" (UniqueName: \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\") pod \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " Nov 01 05:50:48.262275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.262109 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\") pod \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\" (UID: \"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9\") " Nov 01 05:50:48.262550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.262520 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume" pod "3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" (UID: "3bfafc00-3cfa-44a4-8c78-528a18f3a4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:50:48.262687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.262656 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:50:48.264380 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.264357 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8" pod "3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" (UID: "3bfafc00-3cfa-44a4-8c78-528a18f3a4d9"). InnerVolumeSpecName "kube-api-access-l4bm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:50:48.265324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.265308 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:50:48.272392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.272369 4425 generic.go:356] "Generic (PLEG): container finished" podID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" containerID="f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267" exitCode=0 Nov 01 05:50:48.272496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.272409 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" Nov 01 05:50:48.272496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.272450 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" event={"ID":"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9","Type":"ContainerDied","Data":"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267"} Nov 01 05:50:48.272605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.272507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc" event={"ID":"3bfafc00-3cfa-44a4-8c78-528a18f3a4d9","Type":"ContainerDied","Data":"8b6330378104fe585dadf5e29bf81a714f32fa2398ec5cb5b1e330f47c612038"} Nov 01 05:50:48.272605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.272529 4425 scope.go:122] "RemoveContainer" containerID="f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267" Nov 01 05:50:48.284579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.284550 4425 generic.go:356] "Generic (PLEG): container finished" podID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" containerID="fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3" exitCode=0 Nov 01 05:50:48.284743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.284725 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" event={"ID":"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc","Type":"ContainerDied","Data":"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3"} Nov 01 05:50:48.284807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.284778 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" event={"ID":"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc","Type":"ContainerDied","Data":"245167a29d0340e343689485be3f26c93ec86a24b1c0c3a56af36b64aa66f2ef"} Nov 01 05:50:48.284880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.284852 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc" Nov 01 05:50:48.297894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.297868 4425 scope.go:122] "RemoveContainer" containerID="f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267" Nov 01 05:50:48.298285 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:50:48.298243 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267\": container with ID starting with f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267 not found: ID does not exist" containerID="f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267" Nov 01 05:50:48.298421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.298296 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267"} err="failed to get container status \"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267\": rpc error: code = NotFound desc = could not find container \"f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267\": container with ID starting with f6c21a3565d85ed6fd35c8e2a2a23499c9ae31f66469f7842e2e693586e8d267 not found: ID does not exist" Nov 01 05:50:48.298421 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.298321 4425 scope.go:122] "RemoveContainer" containerID="fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3" Nov 01 05:50:48.310128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.310106 4425 scope.go:122] "RemoveContainer" containerID="fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3" Nov 01 05:50:48.310524 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:50:48.310498 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3\": container with ID starting with fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3 not found: ID does not exist" containerID="fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3" Nov 01 05:50:48.310628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.310531 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3"} err="failed to get container status \"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3\": rpc error: code = NotFound desc = could not find container \"fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3\": container with ID starting with fb162adf510076eaedd1857dba8ea15981849a471b0d4422119870afd7f020e3 not found: ID does not exist" Nov 01 05:50:48.363222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.363182 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\") pod \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " Nov 01 05:50:48.363537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.363439 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\" (UniqueName: \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\") pod \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\" (UID: \"2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc\") " Nov 01 05:50:48.363613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.363585 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume" pod "2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" (UID: "2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:50:48.363913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.363894 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4bm8\" (UniqueName: \"kubernetes.io/projected/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9-kube-api-access-l4bm8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:50:48.363991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.363919 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:50:48.366049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.366020 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777" pod "2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" (UID: "2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc"). InnerVolumeSpecName "kube-api-access-sz777". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:50:48.465689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.465597 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz777\" (UniqueName: \"kubernetes.io/projected/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc-kube-api-access-sz777\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:50:48.508706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.508669 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:50:48.515420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.515384 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ad05c6-697dd9486c-56jfc"] Nov 01 05:50:48.707381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.707329 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:50:48.711463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.711429 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ad05c6-6b9b5d9fff-zswjc"] Nov 01 05:50:48.973992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.973960 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" path="/var/lib/kubelet/pods/2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc/volumes" Nov 01 05:50:48.974681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:50:48.974367 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" path="/var/lib/kubelet/pods/3bfafc00-3cfa-44a4-8c78-528a18f3a4d9/volumes" Nov 01 05:50:56.948347 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:50:56.948309 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:51:05.213341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:05.213303 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:51:05.214281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:05.214224 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" containerID="cri-o://cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7" gracePeriod=30 Nov 01 05:51:06.081282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:06.080773 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt"] Nov 01 05:51:06.102074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:06.102032 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb15465b-frpqt"] Nov 01 05:51:06.996229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:06.996182 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7" path="/var/lib/kubelet/pods/fdf51825-e1e8-46fa-bc9a-1cd3790dd2e7/volumes" Nov 01 05:51:07.400997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:07.400966 4425 scope.go:122] "RemoveContainer" containerID="cfa1ba5d1c9d51a0199ce4a98c93b34de0b3e2f0c4422a0d8887e2dcd3222ed7" Nov 01 05:51:07.411734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:07.411717 4425 scope.go:122] "RemoveContainer" containerID="3b0dbdbd22d858cf5bbd25f391c977770837b17f505541d333829a78c82384b1" Nov 01 05:51:07.420641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:07.420624 4425 scope.go:122] "RemoveContainer" containerID="8e9526e4ed716d37031ff8475cf3401093352f01e96cc8d741283ca0b537bcb1" Nov 01 05:51:10.484553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.484525 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:51:10.514048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514015 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514156 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514206 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514296 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:10.514389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514358 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514406 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514434 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514461 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514545 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:10.514667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514632 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.514745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.514722 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\") pod \"721ff6ee-8978-4adc-9bdf-fde443403d74\" (UID: \"721ff6ee-8978-4adc-9bdf-fde443403d74\") " Nov 01 05:51:10.515343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515046 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:10.515343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515075 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:51:10.515343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515210 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:10.515611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515379 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.515611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515402 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.515611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515416 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.515611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515430 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721ff6ee-8978-4adc-9bdf-fde443403d74-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.515611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.515445 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.516747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.516722 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:51:10.517191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.517161 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:51:10.517191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.517185 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:10.517366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.517350 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4" pod "721ff6ee-8978-4adc-9bdf-fde443403d74" (UID: "721ff6ee-8978-4adc-9bdf-fde443403d74"). InnerVolumeSpecName "kube-api-access-dfqv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:10.610823 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.610792 4425 generic.go:356] "Generic (PLEG): container finished" podID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerID="cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7" exitCode=0 Nov 01 05:51:10.611031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.610866 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" Nov 01 05:51:10.611031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.610875 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerDied","Data":"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7"} Nov 01 05:51:10.611031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.610937 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp" event={"ID":"721ff6ee-8978-4adc-9bdf-fde443403d74","Type":"ContainerDied","Data":"ea05451dec4b2f7e7ceed3a0e53935e29e49badd86da85796a1e17381f412acf"} Nov 01 05:51:10.611031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.610951 4425 scope.go:122] "RemoveContainer" containerID="cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7" Nov 01 05:51:10.616187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.616166 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqv4\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-kube-api-access-dfqv4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.616317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.616189 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.616317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.616199 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.616317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.616209 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721ff6ee-8978-4adc-9bdf-fde443403d74-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:10.625874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.625850 4425 scope.go:122] "RemoveContainer" containerID="ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff" Nov 01 05:51:10.636230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.636201 4425 scope.go:122] "RemoveContainer" containerID="cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7" Nov 01 05:51:10.636592 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:51:10.636572 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7\": container with ID starting with cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7 not found: ID does not exist" containerID="cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7" Nov 01 05:51:10.636704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.636606 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7"} err="failed to get container status \"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7\": rpc error: code = NotFound desc = could not find container \"cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7\": container with ID starting with cb5a8fbcbfe5fe8c9e9632fe70109167cebae447ad44552eb1f5e18751b41da7 not found: ID does not exist" Nov 01 05:51:10.636704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.636633 4425 scope.go:122] "RemoveContainer" containerID="ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff" Nov 01 05:51:10.637215 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:51:10.637161 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff\": container with ID starting with ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff not found: ID does not exist" containerID="ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff" Nov 01 05:51:10.637508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.637406 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff"} err="failed to get container status \"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff\": rpc error: code = NotFound desc = could not find container \"ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff\": container with ID starting with ebcdfda19252cb81ba61f8889b9191c2e07e8b902da22ff612f75cff5c93f6ff not found: ID does not exist" Nov 01 05:51:10.852611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.852577 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:51:10.860494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.860464 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ad05c6-openshift-default-5fdb7744c7-469rp"] Nov 01 05:51:10.865339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.865314 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:51:10.865951 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.865933 4425 cpu_manager.go:414] "containerMap: removing container" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.865951 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.865994 4425 cpu_manager.go:414] "containerMap: removing container" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866002 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866019 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" containerName="payload-pre-processing" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866028 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" containerName="payload-pre-processing" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866050 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" containerName="payload-processing" Nov 01 05:51:10.866058 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866057 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" containerName="payload-processing" Nov 01 05:51:10.866502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866225 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:10.866502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866240 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7cf27c-0ea0-4b19-af83-5ea44a4a0ccc" containerName="payload-pre-processing" Nov 01 05:51:10.866502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.866272 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfafc00-3cfa-44a4-8c78-528a18f3a4d9" containerName="payload-processing" Nov 01 05:51:10.871238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.871219 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.920763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.920724 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfbs\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.920917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.920795 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.920917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.920875 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.920997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.920917 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.920997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.920974 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.921081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.921056 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.921143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.921098 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.921273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.921230 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.921334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.921321 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:10.978786 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.978746 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" path="/var/lib/kubelet/pods/721ff6ee-8978-4adc-9bdf-fde443403d74/volumes" Nov 01 05:51:10.979162 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:10.979141 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:51:11.023016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.022979 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023021 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023075 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023111 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023188 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfbs\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023210 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023239 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023287 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023345 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023609 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023568 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023734 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023699 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.023911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023864 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.024059 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.023936 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.024203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.024183 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.025968 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.025943 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.026080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.026068 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.085568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.085534 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.085729 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.085705 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfbs\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gmkbz\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.186532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.186436 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:11.634726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.634697 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:51:11.635471 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:51:11.635441 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a6cbf3_71bf_4b6d_a42c_6daceab6fb75.slice/crio-870faecfdb46b3465644d73b060785bc023f0fd8aee511fd8034b0769fa44149 WatchSource:0}: Error finding container 870faecfdb46b3465644d73b060785bc023f0fd8aee511fd8034b0769fa44149: Status 404 returned error can't find the container with id 870faecfdb46b3465644d73b060785bc023f0fd8aee511fd8034b0769fa44149 Nov 01 05:51:11.638332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.638290 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:11.638422 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.638393 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:11.638462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:11.638441 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:12.630401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:12.630357 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" event={"ID":"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75","Type":"ContainerStarted","Data":"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97"} Nov 01 05:51:12.630605 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:12.630426 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" event={"ID":"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75","Type":"ContainerStarted","Data":"870faecfdb46b3465644d73b060785bc023f0fd8aee511fd8034b0769fa44149"} Nov 01 05:51:12.864189 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:12.861965 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podStartSLOduration=2.861944307 podStartE2EDuration="2.861944307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:12.80254362 +0000 UTC m=+2887.943132553" watchObservedRunningTime="2026-11-01 05:51:12.861944307 +0000 UTC m=+2888.002533233" Nov 01 05:51:13.189080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:13.189034 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:13.191542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:13.191512 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:13.191750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:13.191726 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:14.167841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.167794 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:51:14.168948 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.168926 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="721ff6ee-8978-4adc-9bdf-fde443403d74" containerName="istio-proxy" Nov 01 05:51:14.172793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.172777 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.187194 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.187167 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:14.187333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.187217 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:14.213725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.213700 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:51:14.265337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.264754 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5d585e\"" Nov 01 05:51:14.265337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.264919 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5d585e-dockercfg-22q7f\"" Nov 01 05:51:14.311412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.311162 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsxp\" (UniqueName: \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.311412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.311324 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.322142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.322099 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:51:14.328359 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.328335 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.332565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.332530 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 05:51:14.390193 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.390140 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.396298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.396246 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:51:14.411130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.411087 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 05:51:14.413764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.413737 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsxp\" (UniqueName: \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.414402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.414008 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.414402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.414091 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.414402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.414178 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzhq\" (UniqueName: \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.414938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.414916 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.476488 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.476452 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsxp\" (UniqueName: \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\") pod \"payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.503151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.503124 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:14.516069 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516036 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516181 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516130 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516177 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516269 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516302 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516385 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516338 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.516483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516465 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516566 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzhq\" (UniqueName: \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.516690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516674 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvnv\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516758 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516715 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.516858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.516836 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.517074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.517046 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.578428 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.578396 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzhq\" (UniqueName: \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\") pod \"payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.618858 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.618925 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.618960 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.618980 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.619028 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.619064 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.619280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.619116 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.622777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.619980 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.622777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.620234 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.622777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.620555 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.622777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.620772 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.622777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.622533 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.623277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.623112 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvnv\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.623277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.623170 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.626628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.626578 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.626992 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.626949 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.681407 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.681312 4425 generic.go:356] "Generic (PLEG): container finished" podID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerID="2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d" exitCode=0 Nov 01 05:51:14.681686 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.681378 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerDied","Data":"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d"} Nov 01 05:51:14.682096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.682076 4425 scope.go:122] "RemoveContainer" containerID="2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d" Nov 01 05:51:14.683071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.683021 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.684833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.684448 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:14.684833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.684518 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:14.684833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.684555 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:14.685350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.685001 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvnv\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.693378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.693210 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:14.716347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.716323 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:14.916342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:14.913468 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:51:15.112520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.112493 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:51:15.114289 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:51:15.114236 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db5e27b_ee41_4c02_b0cc_d7e5cd8cea26.slice/crio-ad87fdf96f24709fdaad18d7d874a5ffd1cbb1d366effdb17b03987e797d2b24 WatchSource:0}: Error finding container ad87fdf96f24709fdaad18d7d874a5ffd1cbb1d366effdb17b03987e797d2b24: Status 404 returned error can't find the container with id ad87fdf96f24709fdaad18d7d874a5ffd1cbb1d366effdb17b03987e797d2b24 Nov 01 05:51:15.179462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.179435 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 05:51:15.181022 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:51:15.180977 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be31ef2_1642_4b6b_b69e_c3532095a677.slice/crio-236396752b13960da4a469b7515e282a3039f49d1a0111e84ffb4540f5cfb2b8 WatchSource:0}: Error finding container 236396752b13960da4a469b7515e282a3039f49d1a0111e84ffb4540f5cfb2b8: Status 404 returned error can't find the container with id 236396752b13960da4a469b7515e282a3039f49d1a0111e84ffb4540f5cfb2b8 Nov 01 05:51:15.184000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.183966 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:15.184073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.184041 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:15.184128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.184087 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:15.190637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.190388 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:15.190637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.190431 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:15.707277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.704859 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" event={"ID":"0be31ef2-1642-4b6b-b69e-c3532095a677","Type":"ContainerStarted","Data":"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac"} Nov 01 05:51:15.707277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.704926 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" event={"ID":"0be31ef2-1642-4b6b-b69e-c3532095a677","Type":"ContainerStarted","Data":"236396752b13960da4a469b7515e282a3039f49d1a0111e84ffb4540f5cfb2b8"} Nov 01 05:51:15.707277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.707208 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" event={"ID":"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26","Type":"ContainerStarted","Data":"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017"} Nov 01 05:51:15.707277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.707275 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" event={"ID":"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26","Type":"ContainerStarted","Data":"ad87fdf96f24709fdaad18d7d874a5ffd1cbb1d366effdb17b03987e797d2b24"} Nov 01 05:51:15.707581 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.707499 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:15.709324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.709299 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" event={"ID":"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72","Type":"ContainerStarted","Data":"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04"} Nov 01 05:51:15.709431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.709341 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" event={"ID":"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72","Type":"ContainerStarted","Data":"e5121f069cbe5983bc3dbd2b1cffcb6e6c51ddb071a8c910c814da2e0be9be4a"} Nov 01 05:51:15.709431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.709399 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:15.711586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.711505 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerStarted","Data":"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011"} Nov 01 05:51:15.717321 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.717304 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:15.835172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.835112 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podStartSLOduration=1.8350922079999998 podStartE2EDuration="1.835092208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:15.833284907 +0000 UTC m=+2890.973873829" watchObservedRunningTime="2026-11-01 05:51:15.835092208 +0000 UTC m=+2890.975681129" Nov 01 05:51:15.863281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.857799 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:51:15.985337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:15.984489 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" podStartSLOduration=1.98446993 podStartE2EDuration="1.98446993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:15.983678083 +0000 UTC m=+2891.124267024" watchObservedRunningTime="2026-11-01 05:51:15.98446993 +0000 UTC m=+2891.125058848" Nov 01 05:51:16.139922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.139890 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:51:16.141222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.141195 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 01 05:51:16.141405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.141292 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 01 05:51:16.190277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.187652 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:16.190277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.187713 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:16.257340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.241906 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" podStartSLOduration=2.241887495 podStartE2EDuration="2.241887495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:16.238318904 +0000 UTC m=+2891.378907843" watchObservedRunningTime="2026-11-01 05:51:16.241887495 +0000 UTC m=+2891.382476415" Nov 01 05:51:16.728103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.728065 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 05:51:16.728325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:16.728127 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 05:51:17.140415 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.140383 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 01 05:51:17.140602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.140437 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 01 05:51:17.188280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.187683 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:17.188280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.187744 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:17.720280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.719760 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 05:51:17.720280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:17.719843 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 05:51:18.140565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.140527 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 01 05:51:18.140733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.140594 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 01 05:51:18.189749 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.189707 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gmkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 05:51:18.190050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.190016 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 05:51:18.716971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.716637 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 05:51:18.716971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:18.716692 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 05:51:19.140060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.140023 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 01 05:51:19.140491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.140073 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 01 05:51:19.196612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.196580 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:19.196988 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.196969 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:19.197800 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.197785 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:51:19.717654 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.717610 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 05:51:19.717824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:19.717675 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 05:51:20.140080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:20.140048 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 01 05:51:20.140550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:20.140360 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 01 05:51:20.716971 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:20.716944 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 05:51:20.717239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:20.717217 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 05:51:21.199436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:21.199398 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:51:21.202781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:21.202744 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:51:21.730203 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:21.730170 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:21.730789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:21.730665 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:21.732124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:21.732108 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 05:51:25.972711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:25.972674 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:51:25.977439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:25.977416 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.074667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.074629 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.074874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.074700 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.074874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.074823 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.074874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.074851 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.075043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.074958 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.075107 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.075050 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.075218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.075191 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.075363 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.075345 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.075450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.075430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrklt\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176557 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176623 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrklt\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176673 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176726 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176760 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.176839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.176784 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.177188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.177159 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.177419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.177323 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.177854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.177323 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.177854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.177595 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.177854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.177724 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.179875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.178425 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.179875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.178786 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.179875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.178924 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.180426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.180385 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.181720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.181685 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:51:26.182136 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.182093 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.239098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.239029 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.239098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.239080 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrklt\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\") pod \"e2e-mt-25565930-openshift-default-64f5666649-5crwq\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.290711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.290680 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:26.735196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.735171 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:51:26.736389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.736364 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:51:26.737173 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.737155 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:51:26.738667 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:51:26.738644 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec2eabb_b7fd_4fc2_83a4_cb6340ce8f62.slice/crio-a9260aabc1731d608ae53e7c28f3223ec41263cd7afce85a866558bdf71d18d9 WatchSource:0}: Error finding container a9260aabc1731d608ae53e7c28f3223ec41263cd7afce85a866558bdf71d18d9: Status 404 returned error can't find the container with id a9260aabc1731d608ae53e7c28f3223ec41263cd7afce85a866558bdf71d18d9 Nov 01 05:51:26.741218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.741186 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:26.741319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.741273 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:26.741319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.741308 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:51:26.849520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.849473 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" event={"ID":"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62","Type":"ContainerStarted","Data":"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4"} Nov 01 05:51:26.849690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.849548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" event={"ID":"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62","Type":"ContainerStarted","Data":"a9260aabc1731d608ae53e7c28f3223ec41263cd7afce85a866558bdf71d18d9"} Nov 01 05:51:26.887210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.887159 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:51:26.892227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.892200 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:26.946288 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.945725 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-8fb022cc51dc\"" Nov 01 05:51:26.948370 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.948340 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:51:26.950794 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:51:26.950766 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:51:26.977602 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.977552 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podStartSLOduration=1.977537042 podStartE2EDuration="1.977537042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:26.97337784 +0000 UTC m=+2902.113966786" watchObservedRunningTime="2026-11-01 05:51:26.977537042 +0000 UTC m=+2902.118125961" Nov 01 05:51:26.992570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.992478 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgkm\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:26.992570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.992548 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:26.992810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.992743 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:26.992867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:26.992850 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.093773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.093734 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.093962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.093822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.093962 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.093876 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.094135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.093991 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgkm\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.095279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.094915 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.095279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.095095 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.097587 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.097567 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.157320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.157283 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgkm\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.209761 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.209724 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.291336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.291305 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:27.292683 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.292653 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:27.292822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.292715 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:27.595276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.595220 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:51:27.596687 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:51:27.596662 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50689af_8ebb_4e48_8f1c_0a28f2e5cb56.slice/crio-5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92 WatchSource:0}: Error finding container 5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92: Status 404 returned error can't find the container with id 5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92 Nov 01 05:51:27.858007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.857905 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" event={"ID":"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56","Type":"ContainerStarted","Data":"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645"} Nov 01 05:51:27.858007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.857968 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" event={"ID":"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56","Type":"ContainerStarted","Data":"5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92"} Nov 01 05:51:27.858305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.858008 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:27.979873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:27.979818 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" podStartSLOduration=1.979800228 podStartE2EDuration="1.979800228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:27.979343193 +0000 UTC m=+2903.119932133" watchObservedRunningTime="2026-11-01 05:51:27.979800228 +0000 UTC m=+2903.120389147" Nov 01 05:51:28.291305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:28.291203 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:28.291305 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:28.291283 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:29.291629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:29.291587 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:29.292036 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:29.291653 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:30.291204 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:30.291172 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:30.291397 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:30.291225 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:31.291689 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:31.291654 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:31.292182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:31.291714 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:32.291821 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:32.291789 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:32.292339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:32.291841 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:33.291931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:33.291894 4425 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-5crwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 05:51:33.292354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:33.291949 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 05:51:34.295620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:34.295590 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:34.296122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:34.296034 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:34.296819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:34.296800 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:51:38.875978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:38.875948 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:51:39.049236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:39.049205 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:51:44.049710 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.049638 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" podUID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" containerName="payload-processing" containerID="cri-o://11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2" gracePeriod=25 Nov 01 05:51:44.328408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.328381 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:51:44.353890 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.353865 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\") pod \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " Nov 01 05:51:44.354082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.353943 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\") pod \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " Nov 01 05:51:44.354082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.353998 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\") pod \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " Nov 01 05:51:44.354347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.354323 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume" pod "adf006ad-a63a-4c53-8f9a-8f15a19cef58" (UID: "adf006ad-a63a-4c53-8f9a-8f15a19cef58"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:51:44.354597 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.354580 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:44.356220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.356197 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials" pod "adf006ad-a63a-4c53-8f9a-8f15a19cef58" (UID: "adf006ad-a63a-4c53-8f9a-8f15a19cef58"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:44.356360 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.356338 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw" pod "adf006ad-a63a-4c53-8f9a-8f15a19cef58" (UID: "adf006ad-a63a-4c53-8f9a-8f15a19cef58"). InnerVolumeSpecName "kube-api-access-9nhpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:51:44.455814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.455785 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\") pod \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\" (UID: \"adf006ad-a63a-4c53-8f9a-8f15a19cef58\") " Nov 01 05:51:44.456300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.456271 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume" pod "adf006ad-a63a-4c53-8f9a-8f15a19cef58" (UID: "adf006ad-a63a-4c53-8f9a-8f15a19cef58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:51:44.456457 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.456439 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf006ad-a63a-4c53-8f9a-8f15a19cef58-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:44.456503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.456466 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhpw\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-kube-api-access-9nhpw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:44.456503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.456480 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf006ad-a63a-4c53-8f9a-8f15a19cef58-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:51:44.985574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.985544 4425 generic.go:356] "Generic (PLEG): container finished" podID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" containerID="11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2" exitCode=0 Nov 01 05:51:44.985719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.985610 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" Nov 01 05:51:44.985719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.985630 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" event={"ID":"adf006ad-a63a-4c53-8f9a-8f15a19cef58","Type":"ContainerDied","Data":"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2"} Nov 01 05:51:44.985719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.985701 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r" event={"ID":"adf006ad-a63a-4c53-8f9a-8f15a19cef58","Type":"ContainerDied","Data":"8dbceea94b9323e6b36c233e0106baa22ef3755c2e75b01fc0e9d9420703cd18"} Nov 01 05:51:44.985899 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.985737 4425 scope.go:122] "RemoveContainer" containerID="11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2" Nov 01 05:51:44.995816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.995801 4425 scope.go:122] "RemoveContainer" containerID="11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2" Nov 01 05:51:44.996096 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:51:44.996078 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2\": container with ID starting with 11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2 not found: ID does not exist" containerID="11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2" Nov 01 05:51:44.996149 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:44.996104 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2"} err="failed to get container status \"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2\": rpc error: code = NotFound desc = could not find container \"11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2\": container with ID starting with 11558a00ad641829f66171e47af86080f3d3981d9a3a553b954bd04d15d45eb2 not found: ID does not exist" Nov 01 05:51:45.158035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:45.157997 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:51:45.208765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:45.208732 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b7w5z6r"] Nov 01 05:51:46.973807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:51:46.973730 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" path="/var/lib/kubelet/pods/adf006ad-a63a-4c53-8f9a-8f15a19cef58/volumes" Nov 01 05:51:56.947557 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:51:56.947521 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:52:01.103361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:01.103321 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:52:01.103854 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:01.103552 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" containerID="cri-o://fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286" gracePeriod=30 Nov 01 05:52:06.378663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.378603 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:52:06.463680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463641 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.463858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463694 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.463858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463812 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.463933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463879 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.463933 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463913 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.464016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463950 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.464016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.463980 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.464016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464008 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.464177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464121 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:06.464232 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464188 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\") pod \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\" (UID: \"c403328e-3f76-4ba8-a5d6-a1a4cd810494\") " Nov 01 05:52:06.464446 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464299 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:06.464551 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464524 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:06.464705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464683 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:06.464868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.464848 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:52:06.465234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.465212 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.465334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.465241 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.465334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.465276 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.465334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.465292 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.465334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.465306 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.466298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.466243 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:52:06.466403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.466314 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:52:06.466844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.466819 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:06.466949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.466845 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp" pod "c403328e-3f76-4ba8-a5d6-a1a4cd810494" (UID: "c403328e-3f76-4ba8-a5d6-a1a4cd810494"). InnerVolumeSpecName "kube-api-access-496hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:52:06.566523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.566489 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.566523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.566518 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.566523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.566528 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c403328e-3f76-4ba8-a5d6-a1a4cd810494-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:06.566776 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:06.566537 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496hp\" (UniqueName: \"kubernetes.io/projected/c403328e-3f76-4ba8-a5d6-a1a4cd810494-kube-api-access-496hp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:07.135485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.135450 4425 generic.go:356] "Generic (PLEG): container finished" podID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerID="fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286" exitCode=0 Nov 01 05:52:07.135655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.135495 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" Nov 01 05:52:07.135655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.135539 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" event={"ID":"c403328e-3f76-4ba8-a5d6-a1a4cd810494","Type":"ContainerDied","Data":"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286"} Nov 01 05:52:07.135655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.135594 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw" event={"ID":"c403328e-3f76-4ba8-a5d6-a1a4cd810494","Type":"ContainerDied","Data":"70ed9bf355d0597dd4559297ff29704e2abcb993a93b2217421660a186fb1709"} Nov 01 05:52:07.135655 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.135618 4425 scope.go:122] "RemoveContainer" containerID="fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286" Nov 01 05:52:07.149300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.149235 4425 scope.go:122] "RemoveContainer" containerID="fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286" Nov 01 05:52:07.149674 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:07.149655 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286\": container with ID starting with fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286 not found: ID does not exist" containerID="fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286" Nov 01 05:52:07.149746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.149686 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286"} err="failed to get container status \"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286\": rpc error: code = NotFound desc = could not find container \"fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286\": container with ID starting with fd86761ea2840081aa7d05cf74f12734b5fa84de7d028abf873cb7d1350a9286 not found: ID does not exist" Nov 01 05:52:07.353698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.353660 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:52:07.404513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.404482 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2j4vw"] Nov 01 05:52:07.508324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:07.508295 4425 scope.go:122] "RemoveContainer" containerID="adff3bfa7d9f42f0388dfd915a9ac25043b83c1071bfab9b78ca661f27ae8840" Nov 01 05:52:08.014849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.014817 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7446-9f4e17"] Nov 01 05:52:08.015450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015437 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" Nov 01 05:52:08.015502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015451 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" Nov 01 05:52:08.015502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015489 4425 cpu_manager.go:414] "containerMap: removing container" podUID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" containerName="payload-processing" Nov 01 05:52:08.015502 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015494 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" containerName="payload-processing" Nov 01 05:52:08.015600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015577 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" containerName="istio-proxy" Nov 01 05:52:08.015600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.015594 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf006ad-a63a-4c53-8f9a-8f15a19cef58" containerName="payload-processing" Nov 01 05:52:08.020630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.020609 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:08.072176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.072145 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-9f4e17"] Nov 01 05:52:08.073996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.073969 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wfzxk\"" Nov 01 05:52:08.086702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.086676 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rzc\" (UniqueName: \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\") pod \"test-curl-7446-9f4e17\" (UID: \"a2fa18fd-4500-49d5-8352-11f99efd4393\") " pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:08.188280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.188229 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rzc\" (UniqueName: \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\") pod \"test-curl-7446-9f4e17\" (UID: \"a2fa18fd-4500-49d5-8352-11f99efd4393\") " pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:08.250073 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.250044 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rzc\" (UniqueName: \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\") pod \"test-curl-7446-9f4e17\" (UID: \"a2fa18fd-4500-49d5-8352-11f99efd4393\") " pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:08.333440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.333405 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:08.708972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.708945 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-9f4e17"] Nov 01 05:52:08.710120 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:52:08.710092 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2fa18fd_4500_49d5_8352_11f99efd4393.slice/crio-912d94f65976c6cf7a89314c2a5dba98cf9230676e0128415fdbe7149e391d22 WatchSource:0}: Error finding container 912d94f65976c6cf7a89314c2a5dba98cf9230676e0128415fdbe7149e391d22: Status 404 returned error can't find the container with id 912d94f65976c6cf7a89314c2a5dba98cf9230676e0128415fdbe7149e391d22 Nov 01 05:52:08.712179 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.712164 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 05:52:08.975550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:08.975462 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c403328e-3f76-4ba8-a5d6-a1a4cd810494" path="/var/lib/kubelet/pods/c403328e-3f76-4ba8-a5d6-a1a4cd810494/volumes" Nov 01 05:52:09.154834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:09.154796 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-9f4e17" event={"ID":"a2fa18fd-4500-49d5-8352-11f99efd4393","Type":"ContainerStarted","Data":"912d94f65976c6cf7a89314c2a5dba98cf9230676e0128415fdbe7149e391d22"} Nov 01 05:52:10.002477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.002437 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:52:10.008309 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.008287 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.062302 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.062242 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"routing-overlay\"" Nov 01 05:52:10.065320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.065288 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:52:10.111541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.111475 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c73135\"" Nov 01 05:52:10.111541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.111490 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"kube-root-ca.crt\"" Nov 01 05:52:10.111822 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.111569 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"openshift-service-ca.crt\"" Nov 01 05:52:10.112987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.112962 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"e2e-cmp-legacy-a535ef-provider-cred\"" Nov 01 05:52:10.112987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.112976 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c73135\"/\"payload-processing-external-model-e2e--c4db32b3-dockercfg-v75hh\"" Nov 01 05:52:10.120925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.120895 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.121041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.121019 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.121121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.121070 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.121195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.121178 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w26l\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.223056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.223019 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.223268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.223076 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.223268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.223107 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.223268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.223139 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w26l\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.224020 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.223992 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.224129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.224063 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.226322 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.226295 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.285918 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.285839 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w26l\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\") pod \"payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.323205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.323174 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:10.711137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:10.710947 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:52:10.714505 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:52:10.714473 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a87f3d_fa45_49e7_86f7_65ad07018c34.slice/crio-6d50938978a940b6fea6d5f0ac3bdedce73eaaf190ccdacaabbd05069c8f2efd WatchSource:0}: Error finding container 6d50938978a940b6fea6d5f0ac3bdedce73eaaf190ccdacaabbd05069c8f2efd: Status 404 returned error can't find the container with id 6d50938978a940b6fea6d5f0ac3bdedce73eaaf190ccdacaabbd05069c8f2efd Nov 01 05:52:11.170783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:11.170733 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" event={"ID":"95a87f3d-fa45-49e7-86f7-65ad07018c34","Type":"ContainerStarted","Data":"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44"} Nov 01 05:52:11.171289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:11.170800 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" event={"ID":"95a87f3d-fa45-49e7-86f7-65ad07018c34","Type":"ContainerStarted","Data":"6d50938978a940b6fea6d5f0ac3bdedce73eaaf190ccdacaabbd05069c8f2efd"} Nov 01 05:52:11.171289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:11.170821 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:11.289649 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:11.289592 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" podStartSLOduration=2.289579094 podStartE2EDuration="2.289579094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:11.2892344 +0000 UTC m=+2946.429823351" watchObservedRunningTime="2026-11-01 05:52:11.289579094 +0000 UTC m=+2946.430168012" Nov 01 05:52:12.178295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:12.178226 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-9f4e17" event={"ID":"a2fa18fd-4500-49d5-8352-11f99efd4393","Type":"ContainerStarted","Data":"26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd"} Nov 01 05:52:12.293967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:12.293908 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7446-9f4e17" podStartSLOduration=2.352331206 podStartE2EDuration="5.293893753s" totalImagesPullingTime="2.941562547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:12.293033754 +0000 UTC m=+2947.433622690" watchObservedRunningTime="2026-11-01 05:52:12.293893753 +0000 UTC m=+2947.434482672" Nov 01 05:52:13.816579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:13.816542 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7446-9f4e17"] Nov 01 05:52:13.821161 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:13.821136 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7446-9f4e17"] Nov 01 05:52:14.193049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:14.192956 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7446-9f4e17" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" containerName="test-curl-7446-9f4e17" containerID="cri-o://26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd" gracePeriod=2 Nov 01 05:52:14.245170 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:14.245127 4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7446-9f4e17\" is forbidden: User \"system:node:ip-10-0-141-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-97.us-west-1.compute.internal' and this object" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:15.022520 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:15.022476 4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7446-9f4e17\" is forbidden: User \"system:node:ip-10-0-141-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-97.us-west-1.compute.internal' and this object" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:16.477050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:16.477023 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:16.613684 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:16.613650 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\" (UniqueName: \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\") pod \"a2fa18fd-4500-49d5-8352-11f99efd4393\" (UID: \"a2fa18fd-4500-49d5-8352-11f99efd4393\") " Nov 01 05:52:16.615892 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:16.615861 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc" pod "a2fa18fd-4500-49d5-8352-11f99efd4393" (UID: "a2fa18fd-4500-49d5-8352-11f99efd4393"). InnerVolumeSpecName "kube-api-access-d5rzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:52:16.716044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:16.715967 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rzc\" (UniqueName: \"kubernetes.io/projected/a2fa18fd-4500-49d5-8352-11f99efd4393-kube-api-access-d5rzc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:16.974286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:16.974174 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" path="/var/lib/kubelet/pods/a2fa18fd-4500-49d5-8352-11f99efd4393/volumes" Nov 01 05:52:17.214006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:17.213969 4425 generic.go:356] "Generic (PLEG): container finished" podID="a2fa18fd-4500-49d5-8352-11f99efd4393" containerID="26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd" exitCode=137 Nov 01 05:52:17.214220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:17.214028 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-9f4e17" Nov 01 05:52:17.214220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:17.214070 4425 scope.go:122] "RemoveContainer" containerID="26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd" Nov 01 05:52:17.223630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:17.223611 4425 scope.go:122] "RemoveContainer" containerID="26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd" Nov 01 05:52:17.223938 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:17.223910 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd\": container with ID starting with 26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd not found: ID does not exist" containerID="26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd" Nov 01 05:52:17.224049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:17.223943 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd"} err="failed to get container status \"26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd\": rpc error: code = NotFound desc = could not find container \"26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd\": container with ID starting with 26fe00afcc3cefe3dd4ea45a619a8a4190beee9e8af6fb495b8ed265511965fd not found: ID does not exist" Nov 01 05:52:19.322523 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:19.322490 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:52:22.182393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:22.182365 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:52:26.947494 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:26.947454 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:52:34.347079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.347018 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerName="main" containerID="cri-o://7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee" gracePeriod=45 Nov 01 05:52:34.487516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.487469 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.217:8000/ready\": dial tcp 10.133.0.217:8000: connect: connection refused" Nov 01 05:52:34.628395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.628369 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:52:34.740815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.740784 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.740993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.740833 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.740993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.740870 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\" (UniqueName: \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.740993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.740897 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.741231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741140 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.741231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741139 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:34.741231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741206 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\") pod \"36ad069d-f15c-4882-9029-f5b1eb93baf4\" (UID: \"36ad069d-f15c-4882-9029-f5b1eb93baf4\") " Nov 01 05:52:34.741445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741242 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:34.741445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741415 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:34.741799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741751 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:34.741799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741780 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:34.741922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.741816 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:34.743382 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.743359 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "kube-api-access-d6snn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:52:34.743505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.743385 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:52:34.743604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.743587 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm" pod "36ad069d-f15c-4882-9029-f5b1eb93baf4" (UID: "36ad069d-f15c-4882-9029-f5b1eb93baf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:52:34.843234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.843204 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36ad069d-f15c-4882-9029-f5b1eb93baf4-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:34.843234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.843228 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36ad069d-f15c-4882-9029-f5b1eb93baf4-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:34.843234 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:34.843240 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6snn\" (UniqueName: \"kubernetes.io/projected/36ad069d-f15c-4882-9029-f5b1eb93baf4-kube-api-access-d6snn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:35.329473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.329441 4425 generic.go:356] "Generic (PLEG): container finished" podID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerID="7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee" exitCode=0 Nov 01 05:52:35.329651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.329486 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" Nov 01 05:52:35.329651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.329513 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" event={"ID":"36ad069d-f15c-4882-9029-f5b1eb93baf4","Type":"ContainerDied","Data":"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee"} Nov 01 05:52:35.329651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.329548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn" event={"ID":"36ad069d-f15c-4882-9029-f5b1eb93baf4","Type":"ContainerDied","Data":"bbd4690fd35d6dddc78d88875d736ad4143906b65b7ca94855a9f7d3b9debd88"} Nov 01 05:52:35.329651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.329566 4425 scope.go:122] "RemoveContainer" containerID="7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee" Nov 01 05:52:35.339462 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.339440 4425 scope.go:122] "RemoveContainer" containerID="7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee" Nov 01 05:52:35.339747 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:35.339728 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee\": container with ID starting with 7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee not found: ID does not exist" containerID="7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee" Nov 01 05:52:35.339816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.339760 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee"} err="failed to get container status \"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee\": rpc error: code = NotFound desc = could not find container \"7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee\": container with ID starting with 7abba6b6033d6222af3002175036a69ee8795c4d0427b695c873f632dcfd56ee not found: ID does not exist" Nov 01 05:52:35.496904 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.496841 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:52:35.547814 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:35.547786 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-25565930/e2e-lifecycle-model-25565930-kserve-7d565454d5-qdhmn"] Nov 01 05:52:36.974748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:36.974715 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" path="/var/lib/kubelet/pods/36ad069d-f15c-4882-9029-f5b1eb93baf4/volumes" Nov 01 05:52:51.600151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600118 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb"] Nov 01 05:52:51.600804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600787 4425 cpu_manager.go:414] "containerMap: removing container" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerName="main" Nov 01 05:52:51.600850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600806 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerName="main" Nov 01 05:52:51.600850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600832 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" containerName="test-curl-7446-9f4e17" Nov 01 05:52:51.600850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600837 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" containerName="test-curl-7446-9f4e17" Nov 01 05:52:51.600961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600947 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fa18fd-4500-49d5-8352-11f99efd4393" containerName="test-curl-7446-9f4e17" Nov 01 05:52:51.600995 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.600961 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ad069d-f15c-4882-9029-f5b1eb93baf4" containerName="main" Nov 01 05:52:51.606695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.606674 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.668903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.668867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.669111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.668998 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.669188 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.669114 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4hm\" (UniqueName: \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.711158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.711118 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:52:51.711158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.711123 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:52:51.711474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.711129 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:52:51.711474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.711295 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:52:51.714522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.714488 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb"] Nov 01 05:52:51.770862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.770827 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.771027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.770900 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.771027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.770955 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4hm\" (UniqueName: \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.771725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.771697 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.773473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.773450 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.832932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.832897 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4hm\" (UniqueName: \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\") pod \"maas-api-revoke-sub-225974f4-bjgzb\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:51.921231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:51.921136 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:52.303320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:52.303289 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb"] Nov 01 05:52:52.304374 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:52:52.304333 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f914a3c_9430_4219_97c7_68062db4f6e6.slice/crio-ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb WatchSource:0}: Error finding container ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb: Status 404 returned error can't find the container with id ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb Nov 01 05:52:52.448244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:52.448209 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" event={"ID":"9f914a3c-9430-4219-97c7-68062db4f6e6","Type":"ContainerStarted","Data":"ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb"} Nov 01 05:52:53.457563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:53.457474 4425 generic.go:356] "Generic (PLEG): container finished" podID="9f914a3c-9430-4219-97c7-68062db4f6e6" containerID="f4ff5fb0ffec167316848aa3668e78aa3a830ef321f6cda57efc977753f2b32c" exitCode=0 Nov 01 05:52:53.457982 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:53.457565 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" event={"ID":"9f914a3c-9430-4219-97c7-68062db4f6e6","Type":"ContainerDied","Data":"f4ff5fb0ffec167316848aa3668e78aa3a830ef321f6cda57efc977753f2b32c"} Nov 01 05:52:54.636041 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.636012 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:54.710226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.710191 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\") pod \"9f914a3c-9430-4219-97c7-68062db4f6e6\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " Nov 01 05:52:54.710456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.710248 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\" (UniqueName: \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\") pod \"9f914a3c-9430-4219-97c7-68062db4f6e6\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " Nov 01 05:52:54.710456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.710414 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\") pod \"9f914a3c-9430-4219-97c7-68062db4f6e6\" (UID: \"9f914a3c-9430-4219-97c7-68062db4f6e6\") " Nov 01 05:52:54.710816 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.710788 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca" pod "9f914a3c-9430-4219-97c7-68062db4f6e6" (UID: "9f914a3c-9430-4219-97c7-68062db4f6e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:52:54.710958 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.710909 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:54.712713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.712690 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca" pod "9f914a3c-9430-4219-97c7-68062db4f6e6" (UID: "9f914a3c-9430-4219-97c7-68062db4f6e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:52:54.712713 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.712701 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm" pod "9f914a3c-9430-4219-97c7-68062db4f6e6" (UID: "9f914a3c-9430-4219-97c7-68062db4f6e6"). InnerVolumeSpecName "kube-api-access-tg4hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:52:54.812535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.812499 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f914a3c-9430-4219-97c7-68062db4f6e6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:54.812535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:54.812530 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4hm\" (UniqueName: \"kubernetes.io/projected/9f914a3c-9430-4219-97c7-68062db4f6e6-kube-api-access-tg4hm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:52:55.471840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:55.471808 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" Nov 01 05:52:55.471840 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:55.471822 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb" event={"ID":"9f914a3c-9430-4219-97c7-68062db4f6e6","Type":"ContainerDied","Data":"ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb"} Nov 01 05:52:55.472124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:52:55.471880 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc9a219010247c336fdc8a61eb24d3e5d0389990a335eca4aaefb658ff8a5bb" Nov 01 05:52:56.954641 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:52:56.954596 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:53:05.096057 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.096015 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:05.096668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.096653 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f914a3c-9430-4219-97c7-68062db4f6e6" containerName="revoke-keys" Nov 01 05:53:05.096715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.096670 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f914a3c-9430-4219-97c7-68062db4f6e6" containerName="revoke-keys" Nov 01 05:53:05.096799 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.096789 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f914a3c-9430-4219-97c7-68062db4f6e6" containerName="revoke-keys" Nov 01 05:53:05.100260 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.100232 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.157703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.157666 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:05.179538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.179496 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.179715 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.179558 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.179773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.179742 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgz8\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.179944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.179921 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.281152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281119 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.281409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281161 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.281409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281222 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgz8\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.281409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281344 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.281874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281840 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.282213 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.281996 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.283998 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.283968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.343653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.343611 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgz8\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\") pod \"payload-processing-external-model-ff8dd698-dfp4l\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.413712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.413620 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:05.422553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.422522 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q"] Nov 01 05:53:05.427736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.427701 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.481153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.481126 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:53:05.481315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.481234 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:53:05.481922 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.481897 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:53:05.482040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.481994 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:53:05.483388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.483361 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.483495 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.483469 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.483696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.483675 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wws6c\" (UniqueName: \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.486733 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.486707 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q"] Nov 01 05:53:05.584459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.584416 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wws6c\" (UniqueName: \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.584651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.584506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.584651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.584583 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.585191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.585169 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.587703 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.587652 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.663143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.663105 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wws6c\" (UniqueName: \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\") pod \"maas-api-revoke-sub-0f8f292e-k769q\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.767383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.767298 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:05.800746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:05.800715 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:05.803021 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:05.802995 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38218e0_be0f_4d47_963a_08d069b4f5b5.slice/crio-a40e19aedbf31168aad06fe1cd67757693b46720109684856523b79fdda139b0 WatchSource:0}: Error finding container a40e19aedbf31168aad06fe1cd67757693b46720109684856523b79fdda139b0: Status 404 returned error can't find the container with id a40e19aedbf31168aad06fe1cd67757693b46720109684856523b79fdda139b0 Nov 01 05:53:06.148965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.148938 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q"] Nov 01 05:53:06.149855 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:06.149825 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769c1c0f_7264_4d85_8f8b_c292744e11eb.slice/crio-436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723 WatchSource:0}: Error finding container 436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723: Status 404 returned error can't find the container with id 436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723 Nov 01 05:53:06.554819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.554784 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" event={"ID":"b38218e0-be0f-4d47-963a-08d069b4f5b5","Type":"ContainerStarted","Data":"b6a5b4f7b3db5a0a3cc3e3063346bb146199a0051219f302deb7b34bd8347f4d"} Nov 01 05:53:06.554984 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.554851 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" event={"ID":"b38218e0-be0f-4d47-963a-08d069b4f5b5","Type":"ContainerStarted","Data":"a40e19aedbf31168aad06fe1cd67757693b46720109684856523b79fdda139b0"} Nov 01 05:53:06.555046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.555023 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:06.556708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.556684 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" event={"ID":"769c1c0f-7264-4d85-8f8b-c292744e11eb","Type":"ContainerStarted","Data":"436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723"} Nov 01 05:53:06.674099 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:06.674043 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" podStartSLOduration=1.674027818 podStartE2EDuration="1.674027818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:06.672524156 +0000 UTC m=+3001.813113095" watchObservedRunningTime="2026-11-01 05:53:06.674027818 +0000 UTC m=+3001.814616773" Nov 01 05:53:07.545878 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:07.545797 4425 scope.go:122] "RemoveContainer" containerID="fd4bb27b1d98c81aa1a9aed43c6b873d83f0e31a956cb2f4e72de5d77b9d45f9" Nov 01 05:53:07.565354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:07.565319 4425 generic.go:356] "Generic (PLEG): container finished" podID="769c1c0f-7264-4d85-8f8b-c292744e11eb" containerID="8cdc528b62add34c9ff0bb0bd97e9536b18651cc9ffeaf5b1c2edd412dd89d75" exitCode=0 Nov 01 05:53:07.565481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:07.565371 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" event={"ID":"769c1c0f-7264-4d85-8f8b-c292744e11eb","Type":"ContainerDied","Data":"8cdc528b62add34c9ff0bb0bd97e9536b18651cc9ffeaf5b1c2edd412dd89d75"} Nov 01 05:53:08.733750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.733719 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:08.833916 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.833832 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\" (UniqueName: \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\") pod \"769c1c0f-7264-4d85-8f8b-c292744e11eb\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " Nov 01 05:53:08.834079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.834007 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\") pod \"769c1c0f-7264-4d85-8f8b-c292744e11eb\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " Nov 01 05:53:08.834079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.834032 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\") pod \"769c1c0f-7264-4d85-8f8b-c292744e11eb\" (UID: \"769c1c0f-7264-4d85-8f8b-c292744e11eb\") " Nov 01 05:53:08.834405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.834376 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca" pod "769c1c0f-7264-4d85-8f8b-c292744e11eb" (UID: "769c1c0f-7264-4d85-8f8b-c292744e11eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:08.834662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.834613 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:08.836311 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.836285 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c" pod "769c1c0f-7264-4d85-8f8b-c292744e11eb" (UID: "769c1c0f-7264-4d85-8f8b-c292744e11eb"). InnerVolumeSpecName "kube-api-access-wws6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:08.836432 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.836340 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca" pod "769c1c0f-7264-4d85-8f8b-c292744e11eb" (UID: "769c1c0f-7264-4d85-8f8b-c292744e11eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:53:08.936505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.936465 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/769c1c0f-7264-4d85-8f8b-c292744e11eb-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:08.936505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:08.936500 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wws6c\" (UniqueName: \"kubernetes.io/projected/769c1c0f-7264-4d85-8f8b-c292744e11eb-kube-api-access-wws6c\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:09.580858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:09.580823 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" event={"ID":"769c1c0f-7264-4d85-8f8b-c292744e11eb","Type":"ContainerDied","Data":"436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723"} Nov 01 05:53:09.581050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:09.580872 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="436f22f2dbdbd216b42d56d0d3d77679c7777c9a47662bb5f93305d3707ea723" Nov 01 05:53:09.581050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:09.580836 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q" Nov 01 05:53:10.609269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.609225 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:53:10.609708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.609445 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" containerID="cri-o://282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97" gracePeriod=30 Nov 01 05:53:10.622125 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.622094 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:53:10.622346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.622321 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" containerID="cri-o://780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4" gracePeriod=30 Nov 01 05:53:10.642496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.642456 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:53:10.642672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:10.642654 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" containerID="cri-o://40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d" gracePeriod=30 Nov 01 05:53:15.925101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:15.925073 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:53:15.965482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:15.965451 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:53:15.981643 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:15.981618 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:53:16.062212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062180 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062237 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062297 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062347 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062451 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062486 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062582 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062638 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062637 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.062759 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062707 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\") pod \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\" (UID: \"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75\") " Nov 01 05:53:16.062994 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.062964 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.063246 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.063073 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.063550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.063271 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:16.063550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.063503 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.063795 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.063775 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.063873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.063840 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.064218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.064037 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.064218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.064075 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.065346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.065319 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.065524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.065503 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:53:16.065647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.065612 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "kube-api-access-8zfbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.066300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.066245 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy" pod "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" (UID: "59a6cbf3-71bf-4b6d-a42c-6daceab6fb75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.164757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164674 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.164757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164709 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.164757 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164747 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164765 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164799 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164860 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164914 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164950 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.164992 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165021 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165096 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165163 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165208 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165204 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165240 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165278 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165303 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165371 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.165479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165417 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\") pod \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\" (UID: \"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62\") " Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165703 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\") pod \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\" (UID: \"77c8ba49-5944-4176-97ba-2fc5d20f35a4\") " Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165862 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.165926 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166026 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166040 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166118 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166247 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166280 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166385 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166702 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166728 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166747 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166766 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166782 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166798 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166813 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166828 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166842 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166854 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166866 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166879 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166892 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166906 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfbs\" (UniqueName: \"kubernetes.io/projected/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75-kube-api-access-8zfbs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.167147 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.166922 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169151 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169369 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "kube-api-access-hzjkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169394 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "kube-api-access-jrklt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169408 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169417 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:16.169567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169490 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:53:16.169841 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.169704 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo" pod "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" (UID: "aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:53:16.171030 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.171008 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy" pod "77c8ba49-5944-4176-97ba-2fc5d20f35a4" (UID: "77c8ba49-5944-4176-97ba-2fc5d20f35a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:53:16.268619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268585 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268614 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268624 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjkz\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-kube-api-access-hzjkz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268635 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268644 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrklt\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-kube-api-access-jrklt\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268655 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268664 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.268863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.268674 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c8ba49-5944-4176-97ba-2fc5d20f35a4-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:16.654410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.654375 4425 generic.go:356] "Generic (PLEG): container finished" podID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerID="40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d" exitCode=0 Nov 01 05:53:16.654579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.654465 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" Nov 01 05:53:16.654579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.654465 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerDied","Data":"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d"} Nov 01 05:53:16.654579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.654563 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b" event={"ID":"77c8ba49-5944-4176-97ba-2fc5d20f35a4","Type":"ContainerDied","Data":"f73d24feff8fe2c15137864c1ad87e3d6e7fd9f7a4c09aad9cdc266a5bf06e57"} Nov 01 05:53:16.656524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.654598 4425 scope.go:122] "RemoveContainer" containerID="40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d" Nov 01 05:53:16.656524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.656330 4425 generic.go:356] "Generic (PLEG): container finished" podID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerID="282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97" exitCode=0 Nov 01 05:53:16.656524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.656410 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" Nov 01 05:53:16.656524 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.656461 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" event={"ID":"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75","Type":"ContainerDied","Data":"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97"} Nov 01 05:53:16.656809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.656533 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz" event={"ID":"59a6cbf3-71bf-4b6d-a42c-6daceab6fb75","Type":"ContainerDied","Data":"870faecfdb46b3465644d73b060785bc023f0fd8aee511fd8034b0769fa44149"} Nov 01 05:53:16.658306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.658281 4425 generic.go:356] "Generic (PLEG): container finished" podID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerID="780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4" exitCode=0 Nov 01 05:53:16.658455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.658350 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" event={"ID":"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62","Type":"ContainerDied","Data":"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4"} Nov 01 05:53:16.658455 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.658387 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" event={"ID":"aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62","Type":"ContainerDied","Data":"a9260aabc1731d608ae53e7c28f3223ec41263cd7afce85a866558bdf71d18d9"} Nov 01 05:53:16.658578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.658469 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq" Nov 01 05:53:16.667412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.667388 4425 scope.go:122] "RemoveContainer" containerID="254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249" Nov 01 05:53:16.677379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.677362 4425 scope.go:122] "RemoveContainer" containerID="40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d" Nov 01 05:53:16.677658 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:16.677641 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d\": container with ID starting with 40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d not found: ID does not exist" containerID="40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d" Nov 01 05:53:16.677706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.677667 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d"} err="failed to get container status \"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d\": rpc error: code = NotFound desc = could not find container \"40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d\": container with ID starting with 40346b7d711fdf4f6f1bc8f5c37a44d09db1c0d9776ec43be40ef521563e1d3d not found: ID does not exist" Nov 01 05:53:16.677706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.677685 4425 scope.go:122] "RemoveContainer" containerID="254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249" Nov 01 05:53:16.677927 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:16.677912 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249\": container with ID starting with 254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249 not found: ID does not exist" containerID="254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249" Nov 01 05:53:16.677973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.677930 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249"} err="failed to get container status \"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249\": rpc error: code = NotFound desc = could not find container \"254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249\": container with ID starting with 254fb43e7f35e7e1b5f7131b12f8667259d2fd57b81d34f7268b20183769e249 not found: ID does not exist" Nov 01 05:53:16.677973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.677942 4425 scope.go:122] "RemoveContainer" containerID="282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97" Nov 01 05:53:16.687104 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.687085 4425 scope.go:122] "RemoveContainer" containerID="282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97" Nov 01 05:53:16.687418 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:16.687400 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97\": container with ID starting with 282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97 not found: ID does not exist" containerID="282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97" Nov 01 05:53:16.687513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.687429 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97"} err="failed to get container status \"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97\": rpc error: code = NotFound desc = could not find container \"282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97\": container with ID starting with 282a59fc3da7e3bc036fbc8d5374a88c7823e355cb1b6089aeecd1d435c37f97 not found: ID does not exist" Nov 01 05:53:16.687513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.687453 4425 scope.go:122] "RemoveContainer" containerID="780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4" Nov 01 05:53:16.696788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.696755 4425 scope.go:122] "RemoveContainer" containerID="780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4" Nov 01 05:53:16.697077 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:16.697055 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4\": container with ID starting with 780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4 not found: ID does not exist" containerID="780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4" Nov 01 05:53:16.697143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.697087 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4"} err="failed to get container status \"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4\": rpc error: code = NotFound desc = could not find container \"780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4\": container with ID starting with 780a92625e1906fb67ec5146ec024a2901195a16150e1d776d1cde840de6ebd4 not found: ID does not exist" Nov 01 05:53:16.823693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.823659 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:53:16.874546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.874512 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gmkbz"] Nov 01 05:53:16.941895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.941816 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:53:16.947565 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.947541 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-5crwq"] Nov 01 05:53:16.978727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.978694 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" path="/var/lib/kubelet/pods/59a6cbf3-71bf-4b6d-a42c-6daceab6fb75/volumes" Nov 01 05:53:16.979180 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:16.979167 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" path="/var/lib/kubelet/pods/aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62/volumes" Nov 01 05:53:17.059534 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.059504 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:53:17.062286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.062267 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-rq68b"] Nov 01 05:53:17.554802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.554761 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc"] Nov 01 05:53:17.555419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555404 4425 cpu_manager.go:414] "containerMap: removing container" podUID="769c1c0f-7264-4d85-8f8b-c292744e11eb" containerName="revoke-keys" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555420 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769c1c0f-7264-4d85-8f8b-c292744e11eb" containerName="revoke-keys" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555440 4425 cpu_manager.go:414] "containerMap: removing container" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555446 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555454 4425 cpu_manager.go:414] "containerMap: removing container" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555459 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555470 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" Nov 01 05:53:17.555482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555475 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" Nov 01 05:53:17.555698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555590 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:17.555698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555600 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a6cbf3-71bf-4b6d-a42c-6daceab6fb75" containerName="istio-proxy" Nov 01 05:53:17.555698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555608 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec2eabb-b7fd-4fc2-83a4-cb6340ce8f62" containerName="istio-proxy" Nov 01 05:53:17.555698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555617 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:17.555698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.555625 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="769c1c0f-7264-4d85-8f8b-c292744e11eb" containerName="revoke-keys" Nov 01 05:53:17.560543 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.560522 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.570694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.570661 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:53:17.587420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.587385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqwv\" (UniqueName: \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.587583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.587447 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.587583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.587570 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.614896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.614190 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:53:17.614896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.614342 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:53:17.614896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.614445 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:53:17.614896 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.614774 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:53:17.616747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.616722 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc"] Nov 01 05:53:17.689621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.689573 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqwv\" (UniqueName: \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.689858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.689665 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.689858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.689754 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.690429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.690404 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.692426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.692403 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.753604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.753571 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqwv\" (UniqueName: \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\") pod \"maas-api-revoke-sub-4f234a17-2t2qc\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:17.874976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:17.874936 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:18.253767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.253742 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc"] Nov 01 05:53:18.255400 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:18.255368 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdefdd9_e45c_48e1_8303_bee8d425896a.slice/crio-910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa WatchSource:0}: Error finding container 910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa: Status 404 returned error can't find the container with id 910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa Nov 01 05:53:18.474410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.474378 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:53:18.477336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.477304 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:18.478698 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.478671 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:53:18.676732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.676700 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" event={"ID":"0cdefdd9-e45c-48e1-8303-bee8d425896a","Type":"ContainerStarted","Data":"eab6d4090111d2deccb66b814f2742b8828d29835f12e984e04019e2486f8b64"} Nov 01 05:53:18.676868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.676754 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" event={"ID":"0cdefdd9-e45c-48e1-8303-bee8d425896a","Type":"ContainerStarted","Data":"910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa"} Nov 01 05:53:18.832516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.832460 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" podStartSLOduration=1.832447083 podStartE2EDuration="1.832447083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:18.829421112 +0000 UTC m=+3013.970010035" watchObservedRunningTime="2026-11-01 05:53:18.832447083 +0000 UTC m=+3013.973036001" Nov 01 05:53:18.974675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:18.974642 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" path="/var/lib/kubelet/pods/77c8ba49-5944-4176-97ba-2fc5d20f35a4/volumes" Nov 01 05:53:19.684991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:19.684955 4425 generic.go:356] "Generic (PLEG): container finished" podID="0cdefdd9-e45c-48e1-8303-bee8d425896a" containerID="eab6d4090111d2deccb66b814f2742b8828d29835f12e984e04019e2486f8b64" exitCode=0 Nov 01 05:53:19.685463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:19.685055 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" event={"ID":"0cdefdd9-e45c-48e1-8303-bee8d425896a","Type":"ContainerDied","Data":"eab6d4090111d2deccb66b814f2742b8828d29835f12e984e04019e2486f8b64"} Nov 01 05:53:20.854929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.853518 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:20.958466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.958383 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\") pod \"0cdefdd9-e45c-48e1-8303-bee8d425896a\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " Nov 01 05:53:20.958466 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.958458 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\" (UniqueName: \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\") pod \"0cdefdd9-e45c-48e1-8303-bee8d425896a\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " Nov 01 05:53:20.958680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.958537 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\") pod \"0cdefdd9-e45c-48e1-8303-bee8d425896a\" (UID: \"0cdefdd9-e45c-48e1-8303-bee8d425896a\") " Nov 01 05:53:20.958787 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.958766 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca" pod "0cdefdd9-e45c-48e1-8303-bee8d425896a" (UID: "0cdefdd9-e45c-48e1-8303-bee8d425896a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:20.959006 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.958981 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:20.960801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.960776 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv" pod "0cdefdd9-e45c-48e1-8303-bee8d425896a" (UID: "0cdefdd9-e45c-48e1-8303-bee8d425896a"). InnerVolumeSpecName "kube-api-access-ntqwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:20.960923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:20.960879 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca" pod "0cdefdd9-e45c-48e1-8303-bee8d425896a" (UID: "0cdefdd9-e45c-48e1-8303-bee8d425896a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:53:21.044472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.044429 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 05:53:21.045504 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.045478 4425 cpu_manager.go:414] "containerMap: removing container" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:21.045696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.045680 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c8ba49-5944-4176-97ba-2fc5d20f35a4" containerName="istio-proxy" Nov 01 05:53:21.045879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.045866 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0cdefdd9-e45c-48e1-8303-bee8d425896a" containerName="revoke-keys" Nov 01 05:53:21.045965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.045952 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdefdd9-e45c-48e1-8303-bee8d425896a" containerName="revoke-keys" Nov 01 05:53:21.046236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.046221 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdefdd9-e45c-48e1-8303-bee8d425896a" containerName="revoke-keys" Nov 01 05:53:21.061919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.061885 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqwv\" (UniqueName: \"kubernetes.io/projected/0cdefdd9-e45c-48e1-8303-bee8d425896a-kube-api-access-ntqwv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:21.062231 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.062212 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdefdd9-e45c-48e1-8303-bee8d425896a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:21.078071 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.077176 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.099953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.099912 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 05:53:21.131151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.131097 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d96a66-openshift-default-dockercfg-ftcpn\"" Nov 01 05:53:21.163753 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.163711 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.163997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.163855 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.163997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.163931 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.163997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.163977 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.164216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.164045 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.164216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.164130 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.164216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.164167 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.164216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.164196 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjfh\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.164474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.164305 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.265943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.265844 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.265943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.265917 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.265966 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.265982 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266019 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266063 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266140 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266205 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266171 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266354 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjfh\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266394 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266460 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266593 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266781 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.266952 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.266820 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.268660 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.268639 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.268868 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.268846 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.328680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.328635 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.328881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.328744 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjfh\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\") pod \"e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.411867 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.411804 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:21.701111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.701077 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" Nov 01 05:53:21.701342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.701096 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc" event={"ID":"0cdefdd9-e45c-48e1-8303-bee8d425896a","Type":"ContainerDied","Data":"910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa"} Nov 01 05:53:21.701342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.701170 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="910af7d9de733579c5e4ecfe12899601d64fc26fe2ae945f183ef9bd10cd4bfa" Nov 01 05:53:21.851798 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:21.851756 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc98c712_cfd1_4507_81ba_1c4b519ea1cd.slice/crio-061099da50afaf47ce1488a489beb450c2d659c4c425edbb3295e9ce19b90861 WatchSource:0}: Error finding container 061099da50afaf47ce1488a489beb450c2d659c4c425edbb3295e9ce19b90861: Status 404 returned error can't find the container with id 061099da50afaf47ce1488a489beb450c2d659c4c425edbb3295e9ce19b90861 Nov 01 05:53:21.851949 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.851811 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 05:53:21.854519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.854486 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:53:21.854622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.854565 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:53:21.854622 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:21.854612 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:53:22.709716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:22.709675 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerStarted","Data":"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4"} Nov 01 05:53:22.710183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:22.709735 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerStarted","Data":"061099da50afaf47ce1488a489beb450c2d659c4c425edbb3295e9ce19b90861"} Nov 01 05:53:22.834175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:22.834105 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podStartSLOduration=2.83408699 podStartE2EDuration="2.83408699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:22.830555239 +0000 UTC m=+3017.971144163" watchObservedRunningTime="2026-11-01 05:53:22.83408699 +0000 UTC m=+3017.974675911" Nov 01 05:53:23.412336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.412296 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:23.418490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.418467 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:23.474712 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.474671 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" podUID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" containerName="payload-processing" containerID="cri-o://c1086ca006b0099e1785d2ff0ec04e549ce3ab1fb02876e2a7ab0b2c873b6128" gracePeriod=25 Nov 01 05:53:23.478351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.478298 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" containerID="cri-o://b6a5b4f7b3db5a0a3cc3e3063346bb146199a0051219f302deb7b34bd8347f4d" gracePeriod=25 Nov 01 05:53:23.724358 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.724324 4425 generic.go:356] "Generic (PLEG): container finished" podID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerID="b6a5b4f7b3db5a0a3cc3e3063346bb146199a0051219f302deb7b34bd8347f4d" exitCode=0 Nov 01 05:53:23.725055 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.724441 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" event={"ID":"b38218e0-be0f-4d47-963a-08d069b4f5b5","Type":"ContainerDied","Data":"b6a5b4f7b3db5a0a3cc3e3063346bb146199a0051219f302deb7b34bd8347f4d"} Nov 01 05:53:23.726750 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.726719 4425 generic.go:356] "Generic (PLEG): container finished" podID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" containerID="c1086ca006b0099e1785d2ff0ec04e549ce3ab1fb02876e2a7ab0b2c873b6128" exitCode=0 Nov 01 05:53:23.728182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.728154 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" event={"ID":"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d","Type":"ContainerDied","Data":"c1086ca006b0099e1785d2ff0ec04e549ce3ab1fb02876e2a7ab0b2c873b6128"} Nov 01 05:53:23.728314 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.728223 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:23.729481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.729461 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:53:23.789351 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.789327 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:23.792544 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.792524 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:53:23.899985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.899954 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\") pod \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " Nov 01 05:53:23.900195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\") pod \"b38218e0-be0f-4d47-963a-08d069b4f5b5\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " Nov 01 05:53:23.900195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900116 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\") pod \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " Nov 01 05:53:23.900195 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900145 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\") pod \"b38218e0-be0f-4d47-963a-08d069b4f5b5\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " Nov 01 05:53:23.900393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900212 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\") pod \"b38218e0-be0f-4d47-963a-08d069b4f5b5\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " Nov 01 05:53:23.900393 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900293 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\") pod \"b38218e0-be0f-4d47-963a-08d069b4f5b5\" (UID: \"b38218e0-be0f-4d47-963a-08d069b4f5b5\") " Nov 01 05:53:23.900503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900388 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\") pod \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " Nov 01 05:53:23.900561 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900534 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\") pod \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\" (UID: \"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d\") " Nov 01 05:53:23.900662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900629 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume" pod "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" (UID: "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:23.900662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900642 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume" pod "b38218e0-be0f-4d47-963a-08d069b4f5b5" (UID: "b38218e0-be0f-4d47-963a-08d069b4f5b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:23.900777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900662 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume" pod "b38218e0-be0f-4d47-963a-08d069b4f5b5" (UID: "b38218e0-be0f-4d47-963a-08d069b4f5b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:23.901151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.900975 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume" pod "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" (UID: "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:23.901151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.901077 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:23.901151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.901097 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:23.901151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.901111 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b38218e0-be0f-4d47-963a-08d069b4f5b5-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:23.901151 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.901127 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:23.902926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.902902 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials" pod "b38218e0-be0f-4d47-963a-08d069b4f5b5" (UID: "b38218e0-be0f-4d47-963a-08d069b4f5b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:23.903027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.902943 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials" pod "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" (UID: "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:23.903178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.903147 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8" pod "b38218e0-be0f-4d47-963a-08d069b4f5b5" (UID: "b38218e0-be0f-4d47-963a-08d069b4f5b5"). InnerVolumeSpecName "kube-api-access-vrgz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:23.903329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:23.903312 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d" pod "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" (UID: "9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d"). InnerVolumeSpecName "kube-api-access-ntl5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:24.002290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.002187 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrgz8\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-kube-api-access-vrgz8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:24.002290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.002221 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntl5d\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-kube-api-access-ntl5d\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:24.002290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.002235 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:24.002290 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.002244 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b38218e0-be0f-4d47-963a-08d069b4f5b5-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:24.734668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.734641 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" Nov 01 05:53:24.735182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.734675 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9f596676-2bds7" event={"ID":"9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d","Type":"ContainerDied","Data":"11449111cf655b3508cf03d23c6e1393cf07d2f481ff819e2a1cf67d8945e4e0"} Nov 01 05:53:24.735182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.734748 4425 scope.go:122] "RemoveContainer" containerID="c1086ca006b0099e1785d2ff0ec04e549ce3ab1fb02876e2a7ab0b2c873b6128" Nov 01 05:53:24.736824 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.736787 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" event={"ID":"b38218e0-be0f-4d47-963a-08d069b4f5b5","Type":"ContainerDied","Data":"a40e19aedbf31168aad06fe1cd67757693b46720109684856523b79fdda139b0"} Nov 01 05:53:24.736935 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.736805 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l" Nov 01 05:53:24.745413 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.745392 4425 scope.go:122] "RemoveContainer" containerID="b6a5b4f7b3db5a0a3cc3e3063346bb146199a0051219f302deb7b34bd8347f4d" Nov 01 05:53:24.901946 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.901912 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:53:24.952743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.952707 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9f596676-2bds7"] Nov 01 05:53:24.974362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:24.974331 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" path="/var/lib/kubelet/pods/9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d/volumes" Nov 01 05:53:25.063717 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:25.063690 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:25.067965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:25.067940 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff8dd698-dfp4l"] Nov 01 05:53:26.947594 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:26.947555 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:53:26.975182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:26.975143 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" path="/var/lib/kubelet/pods/b38218e0-be0f-4d47-963a-08d069b4f5b5/volumes" Nov 01 05:53:27.420849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.420811 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c"] Nov 01 05:53:27.421542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421523 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" containerName="payload-processing" Nov 01 05:53:27.421670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421544 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" containerName="payload-processing" Nov 01 05:53:27.421670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421582 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" Nov 01 05:53:27.421670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421589 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" Nov 01 05:53:27.421851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421731 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8188d9-7fa6-4c6f-959c-e6fe7e0fe91d" containerName="payload-processing" Nov 01 05:53:27.421851 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.421748 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38218e0-be0f-4d47-963a-08d069b4f5b5" containerName="payload-processing" Nov 01 05:53:27.426745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.426723 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.481611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.481580 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c"] Nov 01 05:53:27.528221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.528186 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:53:27.528221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.528196 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:53:27.528435 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.528231 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:53:27.530475 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.530456 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:53:27.546425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.546385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkp99\" (UniqueName: \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.546577 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.546471 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.546706 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.546689 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.647700 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.647644 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.647884 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.647797 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkp99\" (UniqueName: \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.647884 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.647847 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.648438 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.648411 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.650411 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.650390 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.710067 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.709992 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkp99\" (UniqueName: \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\") pod \"maas-api-revoke-sub-36b10d23-xrb4c\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:27.740345 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:27.740318 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:28.117697 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:28.117666 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c"] Nov 01 05:53:28.118147 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:28.118125 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9451806a_d97b_4e12_94f4_0ef7a06a8d86.slice/crio-bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327 WatchSource:0}: Error finding container bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327: Status 404 returned error can't find the container with id bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327 Nov 01 05:53:28.769054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:28.768946 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" event={"ID":"9451806a-d97b-4e12-94f4-0ef7a06a8d86","Type":"ContainerStarted","Data":"0d68e235ee98c91a3936fc48294086efa5d10f1f2629611f6898e39b2dbaba45"} Nov 01 05:53:28.769054 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:28.769011 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" event={"ID":"9451806a-d97b-4e12-94f4-0ef7a06a8d86","Type":"ContainerStarted","Data":"bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327"} Nov 01 05:53:28.887536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:28.887475 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" podStartSLOduration=1.887460366 podStartE2EDuration="1.887460366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:28.885872963 +0000 UTC m=+3024.026461959" watchObservedRunningTime="2026-11-01 05:53:28.887460366 +0000 UTC m=+3024.028049285" Nov 01 05:53:29.777668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:29.777632 4425 generic.go:356] "Generic (PLEG): container finished" podID="9451806a-d97b-4e12-94f4-0ef7a06a8d86" containerID="0d68e235ee98c91a3936fc48294086efa5d10f1f2629611f6898e39b2dbaba45" exitCode=0 Nov 01 05:53:29.778094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:29.777722 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" event={"ID":"9451806a-d97b-4e12-94f4-0ef7a06a8d86","Type":"ContainerDied","Data":"0d68e235ee98c91a3936fc48294086efa5d10f1f2629611f6898e39b2dbaba45"} Nov 01 05:53:30.940895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:30.940871 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:31.091408 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.091378 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\") pod \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " Nov 01 05:53:31.091606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.091428 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\" (UniqueName: \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\") pod \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " Nov 01 05:53:31.091606 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.091454 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\") pod \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\" (UID: \"9451806a-d97b-4e12-94f4-0ef7a06a8d86\") " Nov 01 05:53:31.091847 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.091823 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca" pod "9451806a-d97b-4e12-94f4-0ef7a06a8d86" (UID: "9451806a-d97b-4e12-94f4-0ef7a06a8d86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:31.094424 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.094389 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca" pod "9451806a-d97b-4e12-94f4-0ef7a06a8d86" (UID: "9451806a-d97b-4e12-94f4-0ef7a06a8d86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:53:31.094552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.094460 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99" pod "9451806a-d97b-4e12-94f4-0ef7a06a8d86" (UID: "9451806a-d97b-4e12-94f4-0ef7a06a8d86"). InnerVolumeSpecName "kube-api-access-xkp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:31.193336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.193298 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:31.193336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.193329 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkp99\" (UniqueName: \"kubernetes.io/projected/9451806a-d97b-4e12-94f4-0ef7a06a8d86-kube-api-access-xkp99\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:31.193336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.193340 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451806a-d97b-4e12-94f4-0ef7a06a8d86-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:31.793797 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.793766 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" Nov 01 05:53:31.794003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.793810 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c" event={"ID":"9451806a-d97b-4e12-94f4-0ef7a06a8d86","Type":"ContainerDied","Data":"bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327"} Nov 01 05:53:31.794003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:31.793867 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf16ee742d800ba940253734f0270d9453bfbbf60faeacafa4fdc2b6badd327" Nov 01 05:53:32.761662 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.761618 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 05:53:32.762481 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.762462 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9451806a-d97b-4e12-94f4-0ef7a06a8d86" containerName="revoke-keys" Nov 01 05:53:32.762562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.762484 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9451806a-d97b-4e12-94f4-0ef7a06a8d86" containerName="revoke-keys" Nov 01 05:53:32.762623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.762612 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9451806a-d97b-4e12-94f4-0ef7a06a8d86" containerName="revoke-keys" Nov 01 05:53:32.766473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.766454 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:32.822910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.822876 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 05:53:32.870129 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.870092 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d96a66\"" Nov 01 05:53:32.870591 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.870572 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d96a66-dockercfg-zk6wl\"" Nov 01 05:53:32.918215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.918173 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szzh\" (UniqueName: \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:32.918599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.918574 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:32.925191 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.925156 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 05:53:32.930050 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.930026 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:32.967768 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:32.967738 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 05:53:33.020555 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.020469 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szzh\" (UniqueName: \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:33.020704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.020566 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.020704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.020591 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:33.020802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.020703 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfzn5\" (UniqueName: \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.021412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.021390 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:33.082645 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.082616 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szzh\" (UniqueName: \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\") pod \"payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:33.121975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.121937 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.122154 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.122026 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfzn5\" (UniqueName: \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.122747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.122727 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.184015 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.183983 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfzn5\" (UniqueName: \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\") pod \"payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.246583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.246554 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.380540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.380499 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:33.632445 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.632420 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 05:53:33.633261 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:33.633219 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de60f3_08d3_4146_adb6_608eb8cbcbc6.slice/crio-fce9060c791042c9f7ec7c214fd9b12808a964075cc8b24f6363a057efa4e6fc WatchSource:0}: Error finding container fce9060c791042c9f7ec7c214fd9b12808a964075cc8b24f6363a057efa4e6fc: Status 404 returned error can't find the container with id fce9060c791042c9f7ec7c214fd9b12808a964075cc8b24f6363a057efa4e6fc Nov 01 05:53:33.763990 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.763959 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 05:53:33.764846 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:33.764821 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c746c6_d0ec_4333_8bdd_ccb57adfd329.slice/crio-68d3c2ed3374e18a710ebd0d367201c6e242d8285959dd2343d0f02a636fac7d WatchSource:0}: Error finding container 68d3c2ed3374e18a710ebd0d367201c6e242d8285959dd2343d0f02a636fac7d: Status 404 returned error can't find the container with id 68d3c2ed3374e18a710ebd0d367201c6e242d8285959dd2343d0f02a636fac7d Nov 01 05:53:33.809653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.809625 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" event={"ID":"55c746c6-d0ec-4333-8bdd-ccb57adfd329","Type":"ContainerStarted","Data":"68d3c2ed3374e18a710ebd0d367201c6e242d8285959dd2343d0f02a636fac7d"} Nov 01 05:53:33.811553 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.811524 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" event={"ID":"b2de60f3-08d3-4146-adb6-608eb8cbcbc6","Type":"ContainerStarted","Data":"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605"} Nov 01 05:53:33.811687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.811574 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" event={"ID":"b2de60f3-08d3-4146-adb6-608eb8cbcbc6","Type":"ContainerStarted","Data":"fce9060c791042c9f7ec7c214fd9b12808a964075cc8b24f6363a057efa4e6fc"} Nov 01 05:53:33.811687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.811607 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:33.930601 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:33.930484 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" podStartSLOduration=1.930468598 podStartE2EDuration="1.930468598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:33.928467145 +0000 UTC m=+3029.069056064" watchObservedRunningTime="2026-11-01 05:53:33.930468598 +0000 UTC m=+3029.071057518" Nov 01 05:53:34.820013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:34.819973 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" event={"ID":"55c746c6-d0ec-4333-8bdd-ccb57adfd329","Type":"ContainerStarted","Data":"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0"} Nov 01 05:53:34.820471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:34.820081 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:34.941631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:34.941572 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" podStartSLOduration=2.941555635 podStartE2EDuration="2.941555635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:34.937647311 +0000 UTC m=+3030.078236261" watchObservedRunningTime="2026-11-01 05:53:34.941555635 +0000 UTC m=+3030.082144554" Nov 01 05:53:36.796131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.796083 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw"] Nov 01 05:53:36.800152 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.800134 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:36.858008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.857982 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw"] Nov 01 05:53:36.903976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.903944 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:53:36.903976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.903962 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:53:36.903976 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.903963 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:53:36.904223 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.904059 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:53:36.973913 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.973880 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:36.974096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.973926 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:36.974096 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:36.974075 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lxr\" (UniqueName: \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.075333 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.075294 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.075590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.075384 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.075590 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.075527 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lxr\" (UniqueName: \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.076157 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.076096 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.078034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.078013 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.138726 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.138692 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lxr\" (UniqueName: \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\") pod \"maas-api-revoke-sub-121edbdf-kwrbw\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.414654 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.414555 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:37.796575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.796541 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw"] Nov 01 05:53:37.797702 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:37.797675 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474e0cb_34c2_4a07_b3ca_96b7313b9df6.slice/crio-1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0 WatchSource:0}: Error finding container 1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0: Status 404 returned error can't find the container with id 1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0 Nov 01 05:53:37.841289 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:37.841234 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" event={"ID":"4474e0cb-34c2-4a07-b3ca-96b7313b9df6","Type":"ContainerStarted","Data":"1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0"} Nov 01 05:53:38.854943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:38.854904 4425 generic.go:356] "Generic (PLEG): container finished" podID="4474e0cb-34c2-4a07-b3ca-96b7313b9df6" containerID="489ee879a0202d67cdd805b1d4dffdac8156e9525549831af3dab192d291af84" exitCode=0 Nov 01 05:53:38.856141 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:38.856109 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" event={"ID":"4474e0cb-34c2-4a07-b3ca-96b7313b9df6","Type":"ContainerDied","Data":"489ee879a0202d67cdd805b1d4dffdac8156e9525549831af3dab192d291af84"} Nov 01 05:53:39.611279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:39.610463 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:53:40.489281 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.481607 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:40.594699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.594658 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\") pod \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " Nov 01 05:53:40.595537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.595512 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\") pod \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " Nov 01 05:53:40.614739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.595365 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca" pod "4474e0cb-34c2-4a07-b3ca-96b7313b9df6" (UID: "4474e0cb-34c2-4a07-b3ca-96b7313b9df6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:40.614739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.606155 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca" pod "4474e0cb-34c2-4a07-b3ca-96b7313b9df6" (UID: "4474e0cb-34c2-4a07-b3ca-96b7313b9df6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:53:40.615143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.615122 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\" (UniqueName: \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\") pod \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\" (UID: \"4474e0cb-34c2-4a07-b3ca-96b7313b9df6\") " Nov 01 05:53:40.627979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.616225 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:40.628269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.628226 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:40.761315 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.759505 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr" pod "4474e0cb-34c2-4a07-b3ca-96b7313b9df6" (UID: "4474e0cb-34c2-4a07-b3ca-96b7313b9df6"). InnerVolumeSpecName "kube-api-access-c4lxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:40.770538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.762150 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lxr\" (UniqueName: \"kubernetes.io/projected/4474e0cb-34c2-4a07-b3ca-96b7313b9df6-kube-api-access-c4lxr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:40.918619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.904556 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" event={"ID":"4474e0cb-34c2-4a07-b3ca-96b7313b9df6","Type":"ContainerDied","Data":"1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0"} Nov 01 05:53:40.918619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.904624 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0d89398b58f900993f0bd1594748a9797269c775bfbb641806f3976c388da0" Nov 01 05:53:40.918619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:40.904698 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw" Nov 01 05:53:44.611336 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.611240 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" podUID="95a87f3d-fa45-49e7-86f7-65ad07018c34" containerName="payload-processing" containerID="cri-o://42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44" gracePeriod=25 Nov 01 05:53:44.824328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.824295 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 05:53:44.917508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.917471 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:53:44.935403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.935366 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\") pod \"95a87f3d-fa45-49e7-86f7-65ad07018c34\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " Nov 01 05:53:44.935607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.935494 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\") pod \"95a87f3d-fa45-49e7-86f7-65ad07018c34\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " Nov 01 05:53:44.935607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.935533 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\") pod \"95a87f3d-fa45-49e7-86f7-65ad07018c34\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " Nov 01 05:53:44.935607 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.935598 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\") pod \"95a87f3d-fa45-49e7-86f7-65ad07018c34\" (UID: \"95a87f3d-fa45-49e7-86f7-65ad07018c34\") " Nov 01 05:53:44.936001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.935974 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume" pod "95a87f3d-fa45-49e7-86f7-65ad07018c34" (UID: "95a87f3d-fa45-49e7-86f7-65ad07018c34"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:44.936101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.936061 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume" pod "95a87f3d-fa45-49e7-86f7-65ad07018c34" (UID: "95a87f3d-fa45-49e7-86f7-65ad07018c34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:53:44.936169 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.936151 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:44.938324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.938293 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l" pod "95a87f3d-fa45-49e7-86f7-65ad07018c34" (UID: "95a87f3d-fa45-49e7-86f7-65ad07018c34"). InnerVolumeSpecName "kube-api-access-2w26l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:44.938469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.938449 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials" pod "95a87f3d-fa45-49e7-86f7-65ad07018c34" (UID: "95a87f3d-fa45-49e7-86f7-65ad07018c34"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:53:44.946579 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.946547 4425 generic.go:356] "Generic (PLEG): container finished" podID="95a87f3d-fa45-49e7-86f7-65ad07018c34" containerID="42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44" exitCode=0 Nov 01 05:53:44.946739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.946600 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" Nov 01 05:53:44.946739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.946634 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" event={"ID":"95a87f3d-fa45-49e7-86f7-65ad07018c34","Type":"ContainerDied","Data":"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44"} Nov 01 05:53:44.946739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.946716 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl" event={"ID":"95a87f3d-fa45-49e7-86f7-65ad07018c34","Type":"ContainerDied","Data":"6d50938978a940b6fea6d5f0ac3bdedce73eaaf190ccdacaabbd05069c8f2efd"} Nov 01 05:53:44.946739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.946738 4425 scope.go:122] "RemoveContainer" containerID="42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44" Nov 01 05:53:44.982573 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.982538 4425 scope.go:122] "RemoveContainer" containerID="42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44" Nov 01 05:53:44.983206 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:44.983173 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44\": container with ID starting with 42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44 not found: ID does not exist" containerID="42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44" Nov 01 05:53:44.983366 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:44.983219 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44"} err="failed to get container status \"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44\": rpc error: code = NotFound desc = could not find container \"42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44\": container with ID starting with 42b6790319ca01ba7f84aaaedc6e3e7238d6c8eeab1085b547f435a994a16f44 not found: ID does not exist" Nov 01 05:53:45.036975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.036932 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w26l\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-kube-api-access-2w26l\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:45.037230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.037210 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a87f3d-fa45-49e7-86f7-65ad07018c34-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:45.037350 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.037236 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95a87f3d-fa45-49e7-86f7-65ad07018c34-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:53:45.181688 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.181585 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:53:45.184880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.184845 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c73135/payload-processing-external-model-e2e-extmod-a-c73135-7f44gsdpl"] Nov 01 05:53:45.830034 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:45.830002 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 05:53:46.974026 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:46.973984 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a87f3d-fa45-49e7-86f7-65ad07018c34" path="/var/lib/kubelet/pods/95a87f3d-fa45-49e7-86f7-65ad07018c34/volumes" Nov 01 05:53:48.048183 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:48.048150 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx"] Nov 01 05:53:48.098629 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:48.098601 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1be13-2xpbx"] Nov 01 05:53:48.975063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:48.975031 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ced1182-cedf-46f6-bdce-f38f5ab0fecd" path="/var/lib/kubelet/pods/6ced1182-cedf-46f6-bdce-f38f5ab0fecd/volumes" Nov 01 05:53:54.663897 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.663855 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:53:54.664563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664546 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4474e0cb-34c2-4a07-b3ca-96b7313b9df6" containerName="revoke-keys" Nov 01 05:53:54.664616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664564 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4474e0cb-34c2-4a07-b3ca-96b7313b9df6" containerName="revoke-keys" Nov 01 05:53:54.664616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664590 4425 cpu_manager.go:414] "containerMap: removing container" podUID="95a87f3d-fa45-49e7-86f7-65ad07018c34" containerName="payload-processing" Nov 01 05:53:54.664616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664596 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a87f3d-fa45-49e7-86f7-65ad07018c34" containerName="payload-processing" Nov 01 05:53:54.664721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664697 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4474e0cb-34c2-4a07-b3ca-96b7313b9df6" containerName="revoke-keys" Nov 01 05:53:54.664721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.664710 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a87f3d-fa45-49e7-86f7-65ad07018c34" containerName="payload-processing" Nov 01 05:53:54.669536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.669516 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.726185 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.726134 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:53:54.760451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760415 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.760672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760479 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29cr\" (UniqueName: \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.760672 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760633 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.760802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760725 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.760802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760765 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.760999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.760978 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.772507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.772483 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d585e\"/\"kube-root-ca.crt\"" Nov 01 05:53:54.772507 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.772490 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d585e\"/\"openshift-service-ca.crt\"" Nov 01 05:53:54.774102 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.774086 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d585e\"/\"default-dockercfg-pq62g\"" Nov 01 05:53:54.774236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.774221 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d585e\"/\"explicit-model-5d585e-kserve-self-signed-certs\"" Nov 01 05:53:54.862559 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862520 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29cr\" (UniqueName: \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.862763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862575 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.862763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862614 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.862763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862634 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.862763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.862953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.862901 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.863187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.863159 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.863187 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.863176 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.863361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.863211 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.865292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.865272 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.865491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.865473 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.924723 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.924649 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29cr\" (UniqueName: \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\") pod \"explicit-model-5d585e-kserve-69c67ccf99-ds2wm\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:54.981763 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:54.981733 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:53:55.361052 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:55.361020 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:53:55.361852 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:53:55.361818 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode90fb758_8497_46e5_97be_4734ad155330.slice/crio-e2f0bec63604547ce02e6e12956371d5951b8d15e85f739d596917c076e5a31f WatchSource:0}: Error finding container e2f0bec63604547ce02e6e12956371d5951b8d15e85f739d596917c076e5a31f: Status 404 returned error can't find the container with id e2f0bec63604547ce02e6e12956371d5951b8d15e85f739d596917c076e5a31f Nov 01 05:53:56.025829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:56.025791 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" event={"ID":"e90fb758-8497-46e5-97be-4734ad155330","Type":"ContainerStarted","Data":"d6a9159c948c0f98c3a003cff3d65021b2b2531738bf0ef0488035356afd6fdc"} Nov 01 05:53:56.025829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:56.025839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" event={"ID":"e90fb758-8497-46e5-97be-4734ad155330","Type":"ContainerStarted","Data":"e2f0bec63604547ce02e6e12956371d5951b8d15e85f739d596917c076e5a31f"} Nov 01 05:53:56.145447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:56.145392 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" podStartSLOduration=2.145375793 podStartE2EDuration="2.145375793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:56.143774884 +0000 UTC m=+3051.284363806" watchObservedRunningTime="2026-11-01 05:53:56.145375793 +0000 UTC m=+3051.285964753" Nov 01 05:53:56.951756 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:53:56.951717 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:53:56.982121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:53:56.982093 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:04.981989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:04.981944 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:04.994832 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:04.994806 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:05.104567 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:05.104539 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:07.580200 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:07.580165 4425 scope.go:122] "RemoveContainer" containerID="2f742bf3064f78d30fcd1a8467b92744f0cfec9f8a7256a88910d2e5c6dde4ee" Nov 01 05:54:14.128986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.128949 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 05:54:14.134819 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.134790 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.206389 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206350 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.206680 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206658 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.206873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206844 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.207009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206891 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.207009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206944 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.207150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.207077 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g5f\" (UniqueName: \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.207292 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.206571 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 05:54:14.249230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.249198 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"kube-root-ca.crt\"" Nov 01 05:54:14.250480 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.249591 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"default-dockercfg-k2c98\"" Nov 01 05:54:14.250665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.249702 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"openshift-service-ca.crt\"" Nov 01 05:54:14.250783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.249847 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-afc555\"/\"rate-test-model-afc555-kserve-self-signed-certs\"" Nov 01 05:54:14.308769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.308722 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.308974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.308790 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.308974 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.308851 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.308990 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g5f\" (UniqueName: \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.309100 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.309144 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309266 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.309225 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.309289 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.309549 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.309526 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.311849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.311818 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.312959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.312929 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.373276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.372735 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g5f\" (UniqueName: \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\") pod \"rate-test-model-afc555-kserve-59d9d79fbf-zx4lw\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.448828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.448736 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:14.865915 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:14.865881 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 05:54:14.866438 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:54:14.866408 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod883a2f7b_c3c6_466b_89f1_215477871d47.slice/crio-ffc3b9e3095ef270663717b753aa4e3fd8cf8ca0669f8723c2dc2efe6dc77b92 WatchSource:0}: Error finding container ffc3b9e3095ef270663717b753aa4e3fd8cf8ca0669f8723c2dc2efe6dc77b92: Status 404 returned error can't find the container with id ffc3b9e3095ef270663717b753aa4e3fd8cf8ca0669f8723c2dc2efe6dc77b92 Nov 01 05:54:15.231441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:15.231285 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" event={"ID":"883a2f7b-c3c6-466b-89f1-215477871d47","Type":"ContainerStarted","Data":"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987"} Nov 01 05:54:15.231441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:15.231351 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" event={"ID":"883a2f7b-c3c6-466b-89f1-215477871d47","Type":"ContainerStarted","Data":"ffc3b9e3095ef270663717b753aa4e3fd8cf8ca0669f8723c2dc2efe6dc77b92"} Nov 01 05:54:15.382206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:15.382125 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" podStartSLOduration=1.382099051 podStartE2EDuration="1.382099051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:54:15.372814704 +0000 UTC m=+3070.513403639" watchObservedRunningTime="2026-11-01 05:54:15.382099051 +0000 UTC m=+3070.522687969" Nov 01 05:54:15.449118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:15.449040 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:24.449431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:24.449395 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:24.462153 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:24.462129 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:25.307767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:25.307740 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 05:54:26.948673 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:54:26.948630 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:54:30.267720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:30.267688 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:54:45.289159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.289105 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" podUID="e90fb758-8497-46e5-97be-4734ad155330" containerName="main" containerID="cri-o://d6a9159c948c0f98c3a003cff3d65021b2b2531738bf0ef0488035356afd6fdc" gracePeriod=45 Nov 01 05:54:45.442264 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.442205 4425 generic.go:356] "Generic (PLEG): container finished" podID="e90fb758-8497-46e5-97be-4734ad155330" containerID="d6a9159c948c0f98c3a003cff3d65021b2b2531738bf0ef0488035356afd6fdc" exitCode=0 Nov 01 05:54:45.442439 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.442293 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" event={"ID":"e90fb758-8497-46e5-97be-4734ad155330","Type":"ContainerDied","Data":"d6a9159c948c0f98c3a003cff3d65021b2b2531738bf0ef0488035356afd6fdc"} Nov 01 05:54:45.554500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.554430 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:45.666509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666480 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666540 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\" (UniqueName: \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666611 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666642 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666676 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\" (UniqueName: \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666696 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\") pod \"e90fb758-8497-46e5-97be-4734ad155330\" (UID: \"e90fb758-8497-46e5-97be-4734ad155330\") " Nov 01 05:54:45.666985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666797 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:54:45.666985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666943 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:54:45.666985 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.666964 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:54:45.667425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.667327 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:45.667425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.667353 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:45.667425 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.667366 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:45.668947 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.668921 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:54:45.669048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.668969 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "kube-api-access-d29cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:54:45.669048 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.668977 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs" pod "e90fb758-8497-46e5-97be-4734ad155330" (UID: "e90fb758-8497-46e5-97be-4734ad155330"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:54:45.768925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.768890 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29cr\" (UniqueName: \"kubernetes.io/projected/e90fb758-8497-46e5-97be-4734ad155330-kube-api-access-d29cr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:45.768925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.768923 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90fb758-8497-46e5-97be-4734ad155330-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:45.768925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:45.768934 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90fb758-8497-46e5-97be-4734ad155330-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:54:46.450074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.449996 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" Nov 01 05:54:46.450074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.450015 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm" event={"ID":"e90fb758-8497-46e5-97be-4734ad155330","Type":"ContainerDied","Data":"e2f0bec63604547ce02e6e12956371d5951b8d15e85f739d596917c076e5a31f"} Nov 01 05:54:46.450074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.450065 4425 scope.go:122] "RemoveContainer" containerID="d6a9159c948c0f98c3a003cff3d65021b2b2531738bf0ef0488035356afd6fdc" Nov 01 05:54:46.617920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.617885 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:54:46.668930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.668900 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5d585e/explicit-model-5d585e-kserve-69c67ccf99-ds2wm"] Nov 01 05:54:46.974134 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:54:46.974103 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90fb758-8497-46e5-97be-4734ad155330" path="/var/lib/kubelet/pods/e90fb758-8497-46e5-97be-4734ad155330/volumes" Nov 01 05:54:56.949740 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:54:56.949704 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:55:07.637337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:07.637300 4425 scope.go:122] "RemoveContainer" containerID="fba5d470d030798107a0e1d5723e2096324734c10b86e039c4f93ddaf9c3dbac" Nov 01 05:55:14.070789 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:14.070753 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:55:14.180496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:14.180459 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:55:16.546996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.546955 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 05:55:16.547876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.547857 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e90fb758-8497-46e5-97be-4734ad155330" containerName="main" Nov 01 05:55:16.547876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.547877 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90fb758-8497-46e5-97be-4734ad155330" containerName="main" Nov 01 05:55:16.548056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.548044 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90fb758-8497-46e5-97be-4734ad155330" containerName="main" Nov 01 05:55:16.553509 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.553487 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.609324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.609285 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 05:55:16.655062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655034 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"rate-test-model-1c4191-kserve-self-signed-certs\"" Nov 01 05:55:16.655300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655106 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"openshift-service-ca.crt\"" Nov 01 05:55:16.655300 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655169 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"default-dockercfg-nqjr7\"" Nov 01 05:55:16.655844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655825 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.655895 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655855 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.656013 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.655992 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.656062 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.656045 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.656220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.656203 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvqn\" (UniqueName: \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.656335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.656292 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.659604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.659587 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1c4191\"/\"kube-root-ca.crt\"" Nov 01 05:55:16.757447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757408 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757470 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757581 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvqn\" (UniqueName: \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757624 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757734 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757810 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757761 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.757932 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.757865 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.758148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.758112 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.758148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.758137 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.759967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.759940 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.760294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.760246 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.819747 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.819708 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvqn\" (UniqueName: \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\") pod \"rate-test-model-1c4191-kserve-67789c6677-7lgk6\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:16.879769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:16.879735 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:17.263981 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:17.263948 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 05:55:17.264356 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:17.264334 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod811a9a71_fc8f_44d9_9a0b_d6ecf94e8718.slice/crio-e78b0b2a8e200f31fd353404a94d44778adf1258e57706f11a0836247b95cfc5 WatchSource:0}: Error finding container e78b0b2a8e200f31fd353404a94d44778adf1258e57706f11a0836247b95cfc5: Status 404 returned error can't find the container with id e78b0b2a8e200f31fd353404a94d44778adf1258e57706f11a0836247b95cfc5 Nov 01 05:55:17.671973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:17.671929 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" event={"ID":"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718","Type":"ContainerStarted","Data":"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822"} Nov 01 05:55:17.672450 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:17.671986 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" event={"ID":"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718","Type":"ContainerStarted","Data":"e78b0b2a8e200f31fd353404a94d44778adf1258e57706f11a0836247b95cfc5"} Nov 01 05:55:17.791685 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:17.791630 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" podStartSLOduration=1.79161678 podStartE2EDuration="1.79161678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:17.789635533 +0000 UTC m=+3132.930224452" watchObservedRunningTime="2026-11-01 05:55:17.79161678 +0000 UTC m=+3132.932205699" Nov 01 05:55:17.880711 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:17.880675 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:18.373325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.373290 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj"] Nov 01 05:55:18.377499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.377477 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.431031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.430999 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:55:18.434732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.434709 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj"] Nov 01 05:55:18.476624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.476591 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.476802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.476638 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnw5\" (UniqueName: \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.476802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.476760 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.481089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.481068 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:55:18.481220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.481123 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:55:18.481220 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.481135 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:55:18.578197 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.578154 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.578482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.578215 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnw5\" (UniqueName: \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.578482 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.578286 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.578939 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.578909 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.580777 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.580754 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.640340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.640272 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnw5\" (UniqueName: \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\") pod \"maas-api-revoke-sub-ed68a12c-bssrj\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:18.691804 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:18.691775 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:19.072000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.071969 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj"] Nov 01 05:55:19.072156 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.071990 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" podUID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" containerName="payload-processing" containerID="cri-o://07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017" gracePeriod=25 Nov 01 05:55:19.072695 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:19.072629 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa50a9e_d3b3_44bc_bc28_762ac9d270e3.slice/crio-fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282 WatchSource:0}: Error finding container fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282: Status 404 returned error can't find the container with id fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282 Nov 01 05:55:19.181005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.180959 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" podUID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" containerName="payload-pre-processing" containerID="cri-o://4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04" gracePeriod=25 Nov 01 05:55:19.347600 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.347572 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:55:19.387403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.387355 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\" (UniqueName: \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\") pod \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " Nov 01 05:55:19.387403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.387554 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\") pod \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\" (UID: \"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26\") " Nov 01 05:55:19.387403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.387908 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume" pod "6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" (UID: "6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:55:19.388525 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.388211 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:19.390176 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.390148 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq" pod "6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" (UID: "6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26"). InnerVolumeSpecName "kube-api-access-sjzhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:55:19.449427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.449402 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:55:19.489770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.489735 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\" (UniqueName: \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\") pod \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " Nov 01 05:55:19.489923 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.489828 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\") pod \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\" (UID: \"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72\") " Nov 01 05:55:19.490308 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.490281 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume" pod "9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" (UID: "9af0893b-4f4f-4b5e-bcaa-80e7b7871c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:55:19.490414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.490362 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzhq\" (UniqueName: \"kubernetes.io/projected/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26-kube-api-access-sjzhq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:19.492192 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.492162 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp" pod "9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" (UID: "9af0893b-4f4f-4b5e-bcaa-80e7b7871c72"). InnerVolumeSpecName "kube-api-access-zfsxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:55:19.592384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.592307 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:19.592384 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.592342 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsxp\" (UniqueName: \"kubernetes.io/projected/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72-kube-api-access-zfsxp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:19.686882 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.686833 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" event={"ID":"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3","Type":"ContainerStarted","Data":"622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4"} Nov 01 05:55:19.687063 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.686894 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" event={"ID":"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3","Type":"ContainerStarted","Data":"fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282"} Nov 01 05:55:19.688374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.688342 4425 generic.go:356] "Generic (PLEG): container finished" podID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" containerID="07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017" exitCode=0 Nov 01 05:55:19.688511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.688391 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" Nov 01 05:55:19.688511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.688426 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" event={"ID":"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26","Type":"ContainerDied","Data":"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017"} Nov 01 05:55:19.688511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.688472 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl" event={"ID":"6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26","Type":"ContainerDied","Data":"ad87fdf96f24709fdaad18d7d874a5ffd1cbb1d366effdb17b03987e797d2b24"} Nov 01 05:55:19.688511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.688497 4425 scope.go:122] "RemoveContainer" containerID="07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017" Nov 01 05:55:19.689917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.689896 4425 generic.go:356] "Generic (PLEG): container finished" podID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" containerID="4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04" exitCode=0 Nov 01 05:55:19.690016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.689959 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" Nov 01 05:55:19.690016 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.689983 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" event={"ID":"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72","Type":"ContainerDied","Data":"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04"} Nov 01 05:55:19.690145 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.690041 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc" event={"ID":"9af0893b-4f4f-4b5e-bcaa-80e7b7871c72","Type":"ContainerDied","Data":"e5121f069cbe5983bc3dbd2b1cffcb6e6c51ddb071a8c910c814da2e0be9be4a"} Nov 01 05:55:19.702278 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.702053 4425 scope.go:122] "RemoveContainer" containerID="07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017" Nov 01 05:55:19.702540 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:55:19.702370 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017\": container with ID starting with 07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017 not found: ID does not exist" containerID="07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017" Nov 01 05:55:19.702540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.702397 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017"} err="failed to get container status \"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017\": rpc error: code = NotFound desc = could not find container \"07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017\": container with ID starting with 07d9fc79a19854de7dc7441dc3ee0171b9fc8f9ffe96646f870670e9895a7017 not found: ID does not exist" Nov 01 05:55:19.702540 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.702414 4425 scope.go:122] "RemoveContainer" containerID="4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04" Nov 01 05:55:19.711447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.711429 4425 scope.go:122] "RemoveContainer" containerID="4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04" Nov 01 05:55:19.711703 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:55:19.711685 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04\": container with ID starting with 4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04 not found: ID does not exist" containerID="4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04" Nov 01 05:55:19.711754 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.711711 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04"} err="failed to get container status \"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04\": rpc error: code = NotFound desc = could not find container \"4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04\": container with ID starting with 4ab6fdd56322fbb76fd40054938f766db9d36b546d7adc0b399275c23c2baa04 not found: ID does not exist" Nov 01 05:55:19.809324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.809267 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" podStartSLOduration=1.809239967 podStartE2EDuration="1.809239967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:19.807093884 +0000 UTC m=+3134.947682824" watchObservedRunningTime="2026-11-01 05:55:19.809239967 +0000 UTC m=+3134.949828886" Nov 01 05:55:19.922118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.922017 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:55:19.927019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:19.926993 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5d585e-687cb4555c-dgwpl"] Nov 01 05:55:20.041112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.041070 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:55:20.046875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.046845 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5d585e-5746b6dc67-ql7vc"] Nov 01 05:55:20.158268 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:55:20.158226 4425 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa50a9e_d3b3_44bc_bc28_762ac9d270e3.slice/crio-622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa50a9e_d3b3_44bc_bc28_762ac9d270e3.slice/crio-conmon-622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4.scope\": RecentStats: unable to find data in memory cache]" Nov 01 05:55:20.698017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.697981 4425 generic.go:356] "Generic (PLEG): container finished" podID="eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" containerID="622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4" exitCode=0 Nov 01 05:55:20.698222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.698053 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" event={"ID":"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3","Type":"ContainerDied","Data":"622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4"} Nov 01 05:55:20.973829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.973747 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" path="/var/lib/kubelet/pods/6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26/volumes" Nov 01 05:55:20.974202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:20.974185 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" path="/var/lib/kubelet/pods/9af0893b-4f4f-4b5e-bcaa-80e7b7871c72/volumes" Nov 01 05:55:21.867746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.867719 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:21.920334 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.920299 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\") pod \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " Nov 01 05:55:21.920519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.920393 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\") pod \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " Nov 01 05:55:21.920519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.920429 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\" (UniqueName: \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\") pod \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\" (UID: \"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3\") " Nov 01 05:55:21.920771 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.920748 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca" pod "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" (UID: "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:55:21.920876 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.920862 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:21.922879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.922847 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5" pod "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" (UID: "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3"). InnerVolumeSpecName "kube-api-access-8rnw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:55:21.923064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:21.923045 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca" pod "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" (UID: "eaa50a9e-d3b3-44bc-bc28-762ac9d270e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:55:22.022378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:22.022297 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:22.022378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:22.022328 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rnw5\" (UniqueName: \"kubernetes.io/projected/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3-kube-api-access-8rnw5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:55:22.714429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:22.714397 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" Nov 01 05:55:22.714612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:22.714389 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj" event={"ID":"eaa50a9e-d3b3-44bc-bc28-762ac9d270e3","Type":"ContainerDied","Data":"fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282"} Nov 01 05:55:22.714612 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:22.714561 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3e6d01a0288de56e6fc6585d262652deb7d50bfbd0c7dd0f518ae26d7d8282" Nov 01 05:55:26.879986 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:26.879944 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:26.892608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:26.892582 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:26.971669 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:55:26.971632 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:55:27.771447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:27.771411 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 05:55:37.021038 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021003 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:55:37.021630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021616 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" containerName="payload-processing" Nov 01 05:55:37.021675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021631 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" containerName="payload-processing" Nov 01 05:55:37.021675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021671 4425 cpu_manager.go:414] "containerMap: removing container" podUID="eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" containerName="revoke-keys" Nov 01 05:55:37.021741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021676 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" containerName="revoke-keys" Nov 01 05:55:37.021741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021690 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" containerName="payload-pre-processing" Nov 01 05:55:37.021741 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021695 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" containerName="payload-pre-processing" Nov 01 05:55:37.021845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021792 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" containerName="revoke-keys" Nov 01 05:55:37.021845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021799 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af0893b-4f4f-4b5e-bcaa-80e7b7871c72" containerName="payload-pre-processing" Nov 01 05:55:37.021845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.021806 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db5e27b-ee41-4c02-b0cc-d7e5cd8cea26" containerName="payload-processing" Nov 01 05:55:37.025340 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.025324 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.087276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.085441 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-40af1b-openshift-default-dockercfg-f7vr7\"" Nov 01 05:55:37.116613 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116555 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzs9p\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.116833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116628 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.116833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116684 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.116833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116718 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.116833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116772 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.116833 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116802 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.117140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116877 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.117140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116933 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.117140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.116989 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.141402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.141370 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:55:37.218452 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218421 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzs9p\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218469 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218525 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218665 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218552 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218701 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218752 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218837 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218896 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.218975 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.218959 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.219282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.219209 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.219440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.219376 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.219440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.219406 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.219440 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.219379 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.219593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.219504 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.222079 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.222047 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.222229 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.222201 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.280619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.280548 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzs9p\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.280766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.280676 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\") pod \"e2e-noten-40af1b-openshift-default-79966c65b5-lt94f\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.339537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.339501 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:37.776603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.776575 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:55:37.777445 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:37.777411 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea90785_aa21_4ede_80d6_e770c9c84f2e.slice/crio-615bb36c0a72f1c79d678a15e7bc05bffd96d97cb15c71a374218cf8e946e99e WatchSource:0}: Error finding container 615bb36c0a72f1c79d678a15e7bc05bffd96d97cb15c71a374218cf8e946e99e: Status 404 returned error can't find the container with id 615bb36c0a72f1c79d678a15e7bc05bffd96d97cb15c71a374218cf8e946e99e Nov 01 05:55:37.779996 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.779958 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:37.780115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.780038 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:37.780115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.780083 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:37.819526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:37.819493 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" event={"ID":"eea90785-aa21-4ede-80d6-e770c9c84f2e","Type":"ContainerStarted","Data":"615bb36c0a72f1c79d678a15e7bc05bffd96d97cb15c71a374218cf8e946e99e"} Nov 01 05:55:38.827500 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:38.827458 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" event={"ID":"eea90785-aa21-4ede-80d6-e770c9c84f2e","Type":"ContainerStarted","Data":"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94"} Nov 01 05:55:39.007537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:39.007469 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" podStartSLOduration=3.007450947 podStartE2EDuration="3.007450947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:39.004311393 +0000 UTC m=+3154.144900335" watchObservedRunningTime="2026-11-01 05:55:39.007450947 +0000 UTC m=+3154.148039877" Nov 01 05:55:39.340560 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:39.340529 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:39.371066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:39.371032 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:39.834944 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:39.834914 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:39.836031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:39.836007 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:55:45.774513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.774481 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 05:55:45.779497 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.779472 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.842122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.842087 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 05:55:45.857845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.857806 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858018 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858003 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858194 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858295 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858278 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858309 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858337 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858511 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858483 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858578 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.858651 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.858622 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.886463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.886431 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-098e4d-openshift-default-dockercfg-79gwq\"" Nov 01 05:55:45.960668 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.960629 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.960871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.960692 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.960871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.960738 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.960871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.960857 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.960997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961056 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961033 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961064 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961092 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961178 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961161 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961593 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.961858 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.961831 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.962146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.962118 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.962659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.962382 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.962659 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.962631 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.965064 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.965027 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:45.965732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:45.965705 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:46.025653 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.025560 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:46.027536 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.027500 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\") pod \"e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:46.094325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.094290 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:46.550573 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:46.550538 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7344589a_0c6e_4ed8_8075_8e88dd1b51d4.slice/crio-05c9ee58c15a0c79515f9da43afdc1e27b70e54f886f50758c6a776ef1083031 WatchSource:0}: Error finding container 05c9ee58c15a0c79515f9da43afdc1e27b70e54f886f50758c6a776ef1083031: Status 404 returned error can't find the container with id 05c9ee58c15a0c79515f9da43afdc1e27b70e54f886f50758c6a776ef1083031 Nov 01 05:55:46.552362 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.552339 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 05:55:46.553648 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.553601 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:46.553736 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.553699 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:46.553785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.553745 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:46.911919 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.911880 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerStarted","Data":"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea"} Nov 01 05:55:46.912496 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:46.911946 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerStarted","Data":"05c9ee58c15a0c79515f9da43afdc1e27b70e54f886f50758c6a776ef1083031"} Nov 01 05:55:47.035620 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:47.035561 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podStartSLOduration=2.035546851 podStartE2EDuration="2.035546851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:47.034302302 +0000 UTC m=+3162.174891223" watchObservedRunningTime="2026-11-01 05:55:47.035546851 +0000 UTC m=+3162.176135766" Nov 01 05:55:47.095033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:47.095006 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:47.100628 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:47.100600 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:47.920113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:47.920078 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:47.921227 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:47.921204 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:55:48.933514 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.933471 4425 generic.go:356] "Generic (PLEG): container finished" podID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerID="ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4" exitCode=0 Nov 01 05:55:48.935433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.935406 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerDied","Data":"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4"} Nov 01 05:55:48.936167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.936142 4425 scope.go:122] "RemoveContainer" containerID="ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4" Nov 01 05:55:48.938127 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.938096 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:48.938202 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.938175 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:48.938243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:48.938216 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:55:49.978005 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:49.977931 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerStarted","Data":"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98"} Nov 01 05:55:50.413276 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:50.412113 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:55:50.414770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:50.414443 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 01 05:55:50.414770 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:50.414498 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 01 05:55:51.417279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:51.413687 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 01 05:55:51.417279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:51.413752 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 01 05:55:52.304279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:52.303354 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" containerName="main" probeResult="failure" output="Get \"https://10.133.1.0:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 05:55:52.413279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:52.412648 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 01 05:55:52.413279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:52.412726 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 01 05:55:53.416426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.413022 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 01 05:55:53.416426 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.413111 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 01 05:55:53.727793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.727713 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:55:53.808725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.808679 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 05:55:53.813049 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.813025 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:53.902317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.891608 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 05:55:53.902317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.897081 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:53.902317 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.897149 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5x6t\" (UniqueName: \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:53.926405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.926370 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-098e4d-dockercfg-c6fmt\"" Nov 01 05:55:53.926616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.926431 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-098e4d\"" Nov 01 05:55:53.985610 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.985522 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 05:55:53.988828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.988808 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:53.998980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.998954 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:53.999115 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.998989 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5x6t\" (UniqueName: \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:53.999695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:53.999676 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:54.076303 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.073978 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5x6t\" (UniqueName: \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\") pod \"payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:54.078760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.078729 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 05:55:54.100766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.100732 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.100938 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.100913 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xhb\" (UniqueName: \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.135414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.135385 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:54.202513 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.202478 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xhb\" (UniqueName: \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.202719 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.202647 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.203403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.203376 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.271390 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.269605 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xhb\" (UniqueName: \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\") pod \"payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.302869 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.301935 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:54.412863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.412508 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 01 05:55:54.412863 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.412573 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 01 05:55:54.537586 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.537554 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 05:55:54.537955 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:54.537805 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4901c925_1e05_4884_a9a1_e717453b340f.slice/crio-6ccdb30e5098b0d792cf46290165e1a9462e19c40d22cfd6268466682a219dec WatchSource:0}: Error finding container 6ccdb30e5098b0d792cf46290165e1a9462e19c40d22cfd6268466682a219dec: Status 404 returned error can't find the container with id 6ccdb30e5098b0d792cf46290165e1a9462e19c40d22cfd6268466682a219dec Nov 01 05:55:54.704738 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:54.704710 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 05:55:54.704987 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:55:54.704967 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb19815_0b0d_47d7_8721_dbd15eafcf93.slice/crio-bc6a41f3f24919d3d8816d55f68750a3edd974f71df7a3cb614f8413da3263a4 WatchSource:0}: Error finding container bc6a41f3f24919d3d8816d55f68750a3edd974f71df7a3cb614f8413da3263a4: Status 404 returned error can't find the container with id bc6a41f3f24919d3d8816d55f68750a3edd974f71df7a3cb614f8413da3263a4 Nov 01 05:55:55.091527 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.091490 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" event={"ID":"4901c925-1e05-4884-a9a1-e717453b340f","Type":"ContainerStarted","Data":"67b9be07648e94e68844ae32c8eb540ce171cb7b42e89c80794f394129057730"} Nov 01 05:55:55.091727 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.091554 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" event={"ID":"4901c925-1e05-4884-a9a1-e717453b340f","Type":"ContainerStarted","Data":"6ccdb30e5098b0d792cf46290165e1a9462e19c40d22cfd6268466682a219dec"} Nov 01 05:55:55.091783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.091725 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:55:55.093361 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.093338 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" event={"ID":"9eb19815-0b0d-47d7-8721-dbd15eafcf93","Type":"ContainerStarted","Data":"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12"} Nov 01 05:55:55.093505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.093376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" event={"ID":"9eb19815-0b0d-47d7-8721-dbd15eafcf93","Type":"ContainerStarted","Data":"bc6a41f3f24919d3d8816d55f68750a3edd974f71df7a3cb614f8413da3263a4"} Nov 01 05:55:55.093505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.093410 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:55:55.217284 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.217218 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" podStartSLOduration=2.217204455 podStartE2EDuration="2.217204455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:55.214631298 +0000 UTC m=+3170.355220241" watchObservedRunningTime="2026-11-01 05:55:55.217204455 +0000 UTC m=+3170.357793374" Nov 01 05:55:55.334751 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.334683 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" podStartSLOduration=2.334662537 podStartE2EDuration="2.334662537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:55.331733862 +0000 UTC m=+3170.472322793" watchObservedRunningTime="2026-11-01 05:55:55.334662537 +0000 UTC m=+3170.475251458" Nov 01 05:55:55.436410 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.436316 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:55:55.438809 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:55:55.438788 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 05:55:56.948451 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:55:56.948413 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:56:04.385494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:04.385455 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:56:04.386418 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:04.386382 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" podUID="eea90785-aa21-4ede-80d6-e770c9c84f2e" containerName="istio-proxy" containerID="cri-o://e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94" gracePeriod=30 Nov 01 05:56:06.103332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:06.103303 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 05:56:06.103812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:06.103446 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 05:56:09.670530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.670499 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:56:09.797898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.797864 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.797898 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.797901 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.797932 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.797983 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798012 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798049 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798122 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798174 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798159 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:56:09.798550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798205 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798243 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:56:09.798550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798288 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\") pod \"eea90785-aa21-4ede-80d6-e770c9c84f2e\" (UID: \"eea90785-aa21-4ede-80d6-e770c9c84f2e\") " Nov 01 05:56:09.798550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798438 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:56:09.798550 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798508 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:56:09.798782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.798601 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:56:09.799215 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.799189 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.799301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.799216 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.799301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.799231 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.799301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.799248 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.799301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.799280 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea90785-aa21-4ede-80d6-e770c9c84f2e-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.800646 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.800621 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:56:09.800779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.800714 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "kube-api-access-mzs9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:56:09.801035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.801013 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:56:09.801132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.801052 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token" pod "eea90785-aa21-4ede-80d6-e770c9c84f2e" (UID: "eea90785-aa21-4ede-80d6-e770c9c84f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:56:09.901004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.900924 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.901004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.900953 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.901004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.900962 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea90785-aa21-4ede-80d6-e770c9c84f2e-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:09.901004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:09.900971 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzs9p\" (UniqueName: \"kubernetes.io/projected/eea90785-aa21-4ede-80d6-e770c9c84f2e-kube-api-access-mzs9p\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:56:10.197094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.196996 4425 generic.go:356] "Generic (PLEG): container finished" podID="eea90785-aa21-4ede-80d6-e770c9c84f2e" containerID="e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94" exitCode=0 Nov 01 05:56:10.197094 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.197046 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" Nov 01 05:56:10.197324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.197098 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" event={"ID":"eea90785-aa21-4ede-80d6-e770c9c84f2e","Type":"ContainerDied","Data":"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94"} Nov 01 05:56:10.197324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.197161 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f" event={"ID":"eea90785-aa21-4ede-80d6-e770c9c84f2e","Type":"ContainerDied","Data":"615bb36c0a72f1c79d678a15e7bc05bffd96d97cb15c71a374218cf8e946e99e"} Nov 01 05:56:10.197324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.197188 4425 scope.go:122] "RemoveContainer" containerID="e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94" Nov 01 05:56:10.208082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.208061 4425 scope.go:122] "RemoveContainer" containerID="e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94" Nov 01 05:56:10.208416 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:56:10.208393 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94\": container with ID starting with e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94 not found: ID does not exist" containerID="e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94" Nov 01 05:56:10.208491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.208423 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94"} err="failed to get container status \"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94\": rpc error: code = NotFound desc = could not find container \"e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94\": container with ID starting with e2188b5867fc691c61ac2c4457c27f51bcd1d27aa44893fd086c906649531d94 not found: ID does not exist" Nov 01 05:56:10.366347 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.366317 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:56:10.416705 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.416676 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-40af1b-openshift-default-79966c65b5-lt94f"] Nov 01 05:56:10.974678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:10.974643 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea90785-aa21-4ede-80d6-e770c9c84f2e" path="/var/lib/kubelet/pods/eea90785-aa21-4ede-80d6-e770c9c84f2e/volumes" Nov 01 05:56:26.947730 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:56:26.947647 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:56:56.690070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.690032 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 05:56:56.690959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.690935 4425 cpu_manager.go:414] "containerMap: removing container" podUID="eea90785-aa21-4ede-80d6-e770c9c84f2e" containerName="istio-proxy" Nov 01 05:56:56.691070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.690960 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea90785-aa21-4ede-80d6-e770c9c84f2e" containerName="istio-proxy" Nov 01 05:56:56.691130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.691093 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea90785-aa21-4ede-80d6-e770c9c84f2e" containerName="istio-proxy" Nov 01 05:56:56.702342 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.701376 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.730241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.730210 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 05:56:56.778873 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.778843 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-897c87\"" Nov 01 05:56:56.781175 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.781147 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-897c87-dockercfg-j2mvt\"" Nov 01 05:56:56.821031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.820324 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.821631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.821605 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptt8\" (UniqueName: \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.828427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.828370 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 05:56:56.844100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.844069 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:56.877331 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.877298 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 05:56:56.923423 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.923390 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.923604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.923493 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptt8\" (UniqueName: \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.923604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.923563 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr4g\" (UniqueName: \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:56.923604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.923584 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:56.924374 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.924351 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:56.948333 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:56:56.948231 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:56:56.986160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:56.986122 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptt8\" (UniqueName: \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\") pod \"payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:57.016997 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.016971 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:57.025113 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.025091 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr4g\" (UniqueName: \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:57.025206 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.025122 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:57.025850 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.025826 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:57.104082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.104053 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr4g\" (UniqueName: \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\") pod \"payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:57.159306 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.159245 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:57.399343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.399317 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 05:56:57.399464 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:56:57.399359 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b0971a_4f88_4acc_aa4b_51f084de7c70.slice/crio-331923b7d810b528e7d12c87b110f2cad2681c8a1a54a1a3a0dcf2a71e72e43e WatchSource:0}: Error finding container 331923b7d810b528e7d12c87b110f2cad2681c8a1a54a1a3a0dcf2a71e72e43e: Status 404 returned error can't find the container with id 331923b7d810b528e7d12c87b110f2cad2681c8a1a54a1a3a0dcf2a71e72e43e Nov 01 05:56:57.506286 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.506225 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" event={"ID":"e8b0971a-4f88-4acc-aa4b-51f084de7c70","Type":"ContainerStarted","Data":"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8"} Nov 01 05:56:57.506469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.506331 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:56:57.506469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.506351 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" event={"ID":"e8b0971a-4f88-4acc-aa4b-51f084de7c70","Type":"ContainerStarted","Data":"331923b7d810b528e7d12c87b110f2cad2681c8a1a54a1a3a0dcf2a71e72e43e"} Nov 01 05:56:57.545669 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.545645 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 05:56:57.546449 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:56:57.546424 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd96a9f4_051c_4089_aa38_85233c9b7715.slice/crio-a7dfc741efb97c91382cfdaf2cefc3c61b43e49f442957bf40113cb0a76baedd WatchSource:0}: Error finding container a7dfc741efb97c91382cfdaf2cefc3c61b43e49f442957bf40113cb0a76baedd: Status 404 returned error can't find the container with id a7dfc741efb97c91382cfdaf2cefc3c61b43e49f442957bf40113cb0a76baedd Nov 01 05:56:57.626282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:57.626202 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" podStartSLOduration=1.6261879000000001 podStartE2EDuration="1.6261879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:57.623640503 +0000 UTC m=+3232.764229423" watchObservedRunningTime="2026-11-01 05:56:57.6261879 +0000 UTC m=+3232.766776818" Nov 01 05:56:58.514844 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:58.514803 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" event={"ID":"dd96a9f4-051c-4089-aa38-85233c9b7715","Type":"ContainerStarted","Data":"2745b5548e2d93c77a4322c3c6570d82da7d1919a774b9e6abc78178a0744db6"} Nov 01 05:56:58.515383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:58.514861 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" event={"ID":"dd96a9f4-051c-4089-aa38-85233c9b7715","Type":"ContainerStarted","Data":"a7dfc741efb97c91382cfdaf2cefc3c61b43e49f442957bf40113cb0a76baedd"} Nov 01 05:56:58.515383 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:58.514881 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:56:58.633328 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:56:58.633268 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" podStartSLOduration=2.633233153 podStartE2EDuration="2.633233153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:58.632008285 +0000 UTC m=+3233.772597240" watchObservedRunningTime="2026-11-01 05:56:58.633233153 +0000 UTC m=+3233.773822071" Nov 01 05:57:08.517714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:08.517679 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 05:57:09.524268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:09.524219 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 05:57:17.067091 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:17.067052 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:57:17.067670 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:17.067330 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" containerID="cri-o://f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011" gracePeriod=30 Nov 01 05:57:21.202312 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:21.202270 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 05:57:21.202781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:21.202330 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 05:57:22.346505 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.346436 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:57:22.389596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389551 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.389596 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389600 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.389862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389664 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.389862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389831 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.389983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389889 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:57:22.389983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389926 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.389983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389958 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.390146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.389998 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.390224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390140 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.390298 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390241 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\") pod \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\" (UID: \"8b1ea8b8-3f8c-4b9a-993c-b73797171178\") " Nov 01 05:57:22.390365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390329 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:57:22.390365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390337 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:57:22.390626 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390568 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:57:22.390881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390809 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.390881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390846 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.390881 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390862 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.391046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.390880 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.391046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.391017 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:57:22.392599 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.392574 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 05:57:22.392728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.392706 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:57:22.392889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.392860 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "kube-api-access-zzwwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:57:22.392999 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.392933 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo" pod "8b1ea8b8-3f8c-4b9a-993c-b73797171178" (UID: "8b1ea8b8-3f8c-4b9a-993c-b73797171178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 05:57:22.492721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.492685 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.492721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.492718 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.492721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.492730 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwwl\" (UniqueName: \"kubernetes.io/projected/8b1ea8b8-3f8c-4b9a-993c-b73797171178-kube-api-access-zzwwl\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.492969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.492739 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.492969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.492749 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1ea8b8-3f8c-4b9a-993c-b73797171178-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:22.705420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.705327 4425 generic.go:356] "Generic (PLEG): container finished" podID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerID="f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011" exitCode=0 Nov 01 05:57:22.705420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.705384 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" Nov 01 05:57:22.705420 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.705410 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerDied","Data":"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011"} Nov 01 05:57:22.705690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.705472 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt" event={"ID":"8b1ea8b8-3f8c-4b9a-993c-b73797171178","Type":"ContainerDied","Data":"af7bbf056f965d222ddbcc0345b3aa51203690a21055069545a509f24c348292"} Nov 01 05:57:22.705690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.705505 4425 scope.go:122] "RemoveContainer" containerID="f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011" Nov 01 05:57:22.717663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.717640 4425 scope.go:122] "RemoveContainer" containerID="2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d" Nov 01 05:57:22.729007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.728986 4425 scope.go:122] "RemoveContainer" containerID="f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011" Nov 01 05:57:22.729359 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:57:22.729339 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011\": container with ID starting with f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011 not found: ID does not exist" containerID="f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011" Nov 01 05:57:22.729458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.729377 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011"} err="failed to get container status \"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011\": rpc error: code = NotFound desc = could not find container \"f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011\": container with ID starting with f30e1e502dc50feb8dd0dfaa70b121517c29644d8c4aa58f8a9052e425fdb011 not found: ID does not exist" Nov 01 05:57:22.729458 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.729403 4425 scope.go:122] "RemoveContainer" containerID="2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d" Nov 01 05:57:22.729681 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:57:22.729662 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d\": container with ID starting with 2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d not found: ID does not exist" containerID="2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d" Nov 01 05:57:22.729742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.729692 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d"} err="failed to get container status \"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d\": rpc error: code = NotFound desc = could not find container \"2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d\": container with ID starting with 2e7146f4677e31411104eb5d548f19c29de111087e5b2ec483105138afd8984d not found: ID does not exist" Nov 01 05:57:22.876886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.876833 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:57:22.880456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.880432 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-4mgxt"] Nov 01 05:57:22.974367 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:22.974294 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" path="/var/lib/kubelet/pods/8b1ea8b8-3f8c-4b9a-993c-b73797171178/volumes" Nov 01 05:57:24.723338 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.723299 4425 generic.go:356] "Generic (PLEG): container finished" podID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerID="31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea" exitCode=0 Nov 01 05:57:24.723779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.723379 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerDied","Data":"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea"} Nov 01 05:57:24.723826 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.723813 4425 scope.go:122] "RemoveContainer" containerID="31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea" Nov 01 05:57:24.725461 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.725428 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:57:24.725583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.725518 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:57:24.725583 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:24.725567 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 05:57:25.732732 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:25.732699 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerStarted","Data":"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f"} Nov 01 05:57:26.095343 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:26.095300 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:57:26.095635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:26.095509 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 01 05:57:26.095635 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:26.095571 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 01 05:57:26.983187 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:57:26.983139 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:57:27.095667 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:27.095620 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 01 05:57:27.095893 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:27.095680 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 01 05:57:28.095603 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:28.095558 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 01 05:57:28.096140 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:28.095609 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 01 05:57:29.095095 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:29.095061 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 01 05:57:29.095323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:29.095118 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 01 05:57:30.095615 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:30.095579 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Nov 01 05:57:30.096027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:30.095639 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Nov 01 05:57:31.134720 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:31.134674 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:57:31.135714 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:31.135688 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:57:31.137130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:31.137088 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 05:57:51.120521 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.120446 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd"] Nov 01 05:57:51.121060 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.121046 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:57:51.121139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.121061 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:57:51.121139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.121098 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:57:51.121139 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.121105 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:57:51.121243 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.121221 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:57:51.124925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.124906 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.182218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.182189 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd"] Nov 01 05:57:51.205724 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.205695 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.205931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.205733 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q764c\" (UniqueName: \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.205931 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.205862 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.228806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.228772 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:57:51.228979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.228870 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:57:51.228979 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.228876 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:57:51.231433 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.231417 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:57:51.307769 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.307734 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.307967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.307786 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q764c\" (UniqueName: \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.307967 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.307868 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.308580 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.308548 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.310355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.310334 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.369436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.369399 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q764c\" (UniqueName: \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\") pod \"maas-api-revoke-sub-d2a61048-ldrdd\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.439239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.439148 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:51.819277 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.819221 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd"] Nov 01 05:57:51.819443 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:57:51.819423 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3162c323_1a1a_4c3b_afad_8f00415f75cb.slice/crio-3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3 WatchSource:0}: Error finding container 3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3: Status 404 returned error can't find the container with id 3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3 Nov 01 05:57:51.957114 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:51.957068 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" event={"ID":"3162c323-1a1a-4c3b-afad-8f00415f75cb","Type":"ContainerStarted","Data":"3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3"} Nov 01 05:57:52.965639 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:52.965543 4425 generic.go:356] "Generic (PLEG): container finished" podID="3162c323-1a1a-4c3b-afad-8f00415f75cb" containerID="7598e28afe15f90e329aab8919daeb0b195e7ed778ecb394045dfbb168340bdf" exitCode=0 Nov 01 05:57:52.966017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:52.965632 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" event={"ID":"3162c323-1a1a-4c3b-afad-8f00415f75cb","Type":"ContainerDied","Data":"7598e28afe15f90e329aab8919daeb0b195e7ed778ecb394045dfbb168340bdf"} Nov 01 05:57:54.130378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.130346 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:54.245414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.245319 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\" (UniqueName: \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\") pod \"3162c323-1a1a-4c3b-afad-8f00415f75cb\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " Nov 01 05:57:54.245589 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.245576 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\") pod \"3162c323-1a1a-4c3b-afad-8f00415f75cb\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " Nov 01 05:57:54.245702 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.245683 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\") pod \"3162c323-1a1a-4c3b-afad-8f00415f75cb\" (UID: \"3162c323-1a1a-4c3b-afad-8f00415f75cb\") " Nov 01 05:57:54.246074 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.246034 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca" pod "3162c323-1a1a-4c3b-afad-8f00415f75cb" (UID: "3162c323-1a1a-4c3b-afad-8f00415f75cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:57:54.246489 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.246447 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:54.248272 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.248226 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c" pod "3162c323-1a1a-4c3b-afad-8f00415f75cb" (UID: "3162c323-1a1a-4c3b-afad-8f00415f75cb"). InnerVolumeSpecName "kube-api-access-q764c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:57:54.249760 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.249730 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca" pod "3162c323-1a1a-4c3b-afad-8f00415f75cb" (UID: "3162c323-1a1a-4c3b-afad-8f00415f75cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:57:54.347993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.347957 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q764c\" (UniqueName: \"kubernetes.io/projected/3162c323-1a1a-4c3b-afad-8f00415f75cb-kube-api-access-q764c\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:54.347993 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.347988 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3162c323-1a1a-4c3b-afad-8f00415f75cb-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:57:54.980624 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.980591 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" Nov 01 05:57:54.980807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.980621 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd" event={"ID":"3162c323-1a1a-4c3b-afad-8f00415f75cb","Type":"ContainerDied","Data":"3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3"} Nov 01 05:57:54.980807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:54.980671 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fdde75e2ce5db520f3257c7066c09072a4b30b8388cab880a29df4539042ef3" Nov 01 05:57:55.095429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:55.095394 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb"] Nov 01 05:57:55.098681 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:55.098658 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225974f4-bjgzb"] Nov 01 05:57:56.947960 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:57:56.947923 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:57:56.975391 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:57:56.975363 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f914a3c-9430-4219-97c7-68062db4f6e6" path="/var/lib/kubelet/pods/9f914a3c-9430-4219-97c7-68062db4f6e6/volumes" Nov 01 05:58:01.808682 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.808645 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4"] Nov 01 05:58:01.809395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.809376 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3162c323-1a1a-4c3b-afad-8f00415f75cb" containerName="revoke-keys" Nov 01 05:58:01.809395 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.809393 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3162c323-1a1a-4c3b-afad-8f00415f75cb" containerName="revoke-keys" Nov 01 05:58:01.809575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.809499 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1ea8b8-3f8c-4b9a-993c-b73797171178" containerName="istio-proxy" Nov 01 05:58:01.809575 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.809515 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3162c323-1a1a-4c3b-afad-8f00415f75cb" containerName="revoke-keys" Nov 01 05:58:01.813699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.813671 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:01.870109 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.870078 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4"] Nov 01 05:58:01.916637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.916588 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:58:01.916637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.916637 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:58:01.917242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.917223 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:58:01.917498 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.917476 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:58:01.944811 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.944770 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:01.945000 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.944965 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:01.945066 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:01.945013 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bk9\" (UniqueName: \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.046563 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.046531 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.046766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.046570 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bk9\" (UniqueName: \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.046766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.046702 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.047318 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.047297 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.049081 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.049050 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.108973 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.108941 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bk9\" (UniqueName: \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\") pod \"maas-api-revoke-sub-6749c390-4vct4\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.128911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.128878 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:02.582633 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:02.582604 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4"] Nov 01 05:58:02.583152 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:02.583121 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6957ee8_4aa6_44a9_8db4_1ae6b0892119.slice/crio-9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f WatchSource:0}: Error finding container 9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f: Status 404 returned error can't find the container with id 9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f Nov 01 05:58:03.036027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:03.035990 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" event={"ID":"a6957ee8-4aa6-44a9-8db4-1ae6b0892119","Type":"ContainerStarted","Data":"98bdcbd56debc8b499ece90546966d94030d6f186e0e67726a8e658d2eaae4c5"} Nov 01 05:58:03.036453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:03.036051 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" event={"ID":"a6957ee8-4aa6-44a9-8db4-1ae6b0892119","Type":"ContainerStarted","Data":"9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f"} Nov 01 05:58:03.156783 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:03.156679 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" podStartSLOduration=2.156661874 podStartE2EDuration="2.156661874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:03.153199094 +0000 UTC m=+3298.293788015" watchObservedRunningTime="2026-11-01 05:58:03.156661874 +0000 UTC m=+3298.297250793" Nov 01 05:58:04.051921 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:04.051832 4425 generic.go:356] "Generic (PLEG): container finished" podID="a6957ee8-4aa6-44a9-8db4-1ae6b0892119" containerID="98bdcbd56debc8b499ece90546966d94030d6f186e0e67726a8e658d2eaae4c5" exitCode=0 Nov 01 05:58:04.052326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:04.051922 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" event={"ID":"a6957ee8-4aa6-44a9-8db4-1ae6b0892119","Type":"ContainerDied","Data":"98bdcbd56debc8b499ece90546966d94030d6f186e0e67726a8e658d2eaae4c5"} Nov 01 05:58:05.216167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.216142 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:05.324937 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.324905 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\") pod \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " Nov 01 05:58:05.325171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.325114 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\") pod \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " Nov 01 05:58:05.325171 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.325162 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\" (UniqueName: \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\") pod \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\" (UID: \"a6957ee8-4aa6-44a9-8db4-1ae6b0892119\") " Nov 01 05:58:05.325880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.325562 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca" pod "a6957ee8-4aa6-44a9-8db4-1ae6b0892119" (UID: "a6957ee8-4aa6-44a9-8db4-1ae6b0892119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:58:05.325880 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.325850 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:05.327845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.327811 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca" pod "a6957ee8-4aa6-44a9-8db4-1ae6b0892119" (UID: "a6957ee8-4aa6-44a9-8db4-1ae6b0892119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:58:05.329571 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.329271 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9" pod "a6957ee8-4aa6-44a9-8db4-1ae6b0892119" (UID: "a6957ee8-4aa6-44a9-8db4-1ae6b0892119"). InnerVolumeSpecName "kube-api-access-s2bk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:58:05.426837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.426804 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bk9\" (UniqueName: \"kubernetes.io/projected/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-kube-api-access-s2bk9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:05.426837 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:05.426839 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6957ee8-4aa6-44a9-8db4-1ae6b0892119-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:06.066483 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:06.066448 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" Nov 01 05:58:06.066695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:06.066490 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4" event={"ID":"a6957ee8-4aa6-44a9-8db4-1ae6b0892119","Type":"ContainerDied","Data":"9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f"} Nov 01 05:58:06.066695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:06.066545 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2b4d1563b515c2c091e30511631230f17fa538ca2fc9e9070c6a092f19dd1f" Nov 01 05:58:09.048828 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:09.048793 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q"] Nov 01 05:58:09.099319 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:09.099291 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f292e-k769q"] Nov 01 05:58:10.974027 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:10.973983 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769c1c0f-7264-4d85-8f8b-c292744e11eb" path="/var/lib/kubelet/pods/769c1c0f-7264-4d85-8f8b-c292744e11eb/volumes" Nov 01 05:58:19.069032 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.068992 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp"] Nov 01 05:58:19.069692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.069675 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a6957ee8-4aa6-44a9-8db4-1ae6b0892119" containerName="revoke-keys" Nov 01 05:58:19.069748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.069693 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6957ee8-4aa6-44a9-8db4-1ae6b0892119" containerName="revoke-keys" Nov 01 05:58:19.069802 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.069793 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6957ee8-4aa6-44a9-8db4-1ae6b0892119" containerName="revoke-keys" Nov 01 05:58:19.074785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.074769 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.130456 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.130425 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp"] Nov 01 05:58:19.176815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.176780 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:58:19.176815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.176803 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:58:19.176815 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.176818 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:58:19.178664 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.178641 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:58:19.216409 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.216377 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.216608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.216417 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hht27\" (UniqueName: \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.216696 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.216671 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.328335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.328027 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.328335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.328161 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.328335 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.328187 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hht27\" (UniqueName: \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.329503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.329471 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.336782 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.336753 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.390835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.390797 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hht27\" (UniqueName: \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\") pod \"maas-api-revoke-sub-a1607c6b-d6khp\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:19.689430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:19.689335 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:20.069392 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.069363 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp"] Nov 01 05:58:20.070569 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:20.070546 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a62c2_2057_4e6a_9810_b9562441e759.slice/crio-d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436 WatchSource:0}: Error finding container d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436: Status 404 returned error can't find the container with id d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436 Nov 01 05:58:20.173983 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.173944 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" event={"ID":"208a62c2-2057-4e6a-9810-b9562441e759","Type":"ContainerStarted","Data":"d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436"} Nov 01 05:58:20.395917 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.395828 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 05:58:20.401908 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.401881 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.455728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.455700 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 05:58:20.501326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.501280 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"openshift-service-ca.crt\"" Nov 01 05:58:20.501532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.501329 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d96a66\"" Nov 01 05:58:20.501532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.501359 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"routing-overlay\"" Nov 01 05:58:20.501532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.501394 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"kube-root-ca.crt\"" Nov 01 05:58:20.501532 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.501412 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"agc-cred-c616ffa3b96d\"" Nov 01 05:58:20.508236 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.508214 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d96a66\"/\"payload-processing-external-model-e2e--38082533-dockercfg-nl9pp\"" Nov 01 05:58:20.545969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.545941 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vxm\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.546130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.545986 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.546230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.546122 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.546230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.546160 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.648044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.647953 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vxm\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.648044 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.648010 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.648316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.648108 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.648316 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.648145 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.648807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.648784 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.649082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.648960 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.650965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.650938 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.709690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.709662 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vxm\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\") pod \"payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:20.718593 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:20.718563 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:21.048911 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.048831 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc"] Nov 01 05:58:21.053008 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.052983 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f234a17-2t2qc"] Nov 01 05:58:21.101283 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.101232 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 05:58:21.139218 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:21.139186 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eaea69_89fa_404f_8c45_e7a07aa05eb2.slice/crio-2c1f44b3c9044f350333fc02fffa8e6193c2c82e7cb89a9c6a783bff76ad0fe0 WatchSource:0}: Error finding container 2c1f44b3c9044f350333fc02fffa8e6193c2c82e7cb89a9c6a783bff76ad0fe0: Status 404 returned error can't find the container with id 2c1f44b3c9044f350333fc02fffa8e6193c2c82e7cb89a9c6a783bff76ad0fe0 Nov 01 05:58:21.182485 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.182452 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" event={"ID":"208a62c2-2057-4e6a-9810-b9562441e759","Type":"ContainerStarted","Data":"758e66859e06a16fd3bdaf51c8ad85f0c05a9a1c3939eab8efb7f021026aabc1"} Nov 01 05:58:21.183860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.183832 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" event={"ID":"33eaea69-89fa-404f-8c45-e7a07aa05eb2","Type":"ContainerStarted","Data":"2c1f44b3c9044f350333fc02fffa8e6193c2c82e7cb89a9c6a783bff76ad0fe0"} Nov 01 05:58:21.300807 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:21.300684 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" podStartSLOduration=2.300666763 podStartE2EDuration="2.300666763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:21.298573233 +0000 UTC m=+3316.439162152" watchObservedRunningTime="2026-11-01 05:58:21.300666763 +0000 UTC m=+3316.441255685" Nov 01 05:58:22.192631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.192588 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" event={"ID":"33eaea69-89fa-404f-8c45-e7a07aa05eb2","Type":"ContainerStarted","Data":"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364"} Nov 01 05:58:22.193142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.192691 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:22.194313 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.194287 4425 generic.go:356] "Generic (PLEG): container finished" podID="208a62c2-2057-4e6a-9810-b9562441e759" containerID="758e66859e06a16fd3bdaf51c8ad85f0c05a9a1c3939eab8efb7f021026aabc1" exitCode=0 Nov 01 05:58:22.194417 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.194345 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" event={"ID":"208a62c2-2057-4e6a-9810-b9562441e759","Type":"ContainerDied","Data":"758e66859e06a16fd3bdaf51c8ad85f0c05a9a1c3939eab8efb7f021026aabc1"} Nov 01 05:58:22.310825 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.310775 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" podStartSLOduration=2.310761587 podStartE2EDuration="2.310761587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:22.310089428 +0000 UTC m=+3317.450678359" watchObservedRunningTime="2026-11-01 05:58:22.310761587 +0000 UTC m=+3317.451350506" Nov 01 05:58:22.975479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:22.975434 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdefdd9-e45c-48e1-8303-bee8d425896a" path="/var/lib/kubelet/pods/0cdefdd9-e45c-48e1-8303-bee8d425896a/volumes" Nov 01 05:58:23.364671 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.364643 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:23.381242 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.381217 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\") pod \"208a62c2-2057-4e6a-9810-b9562441e759\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " Nov 01 05:58:23.381355 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.381333 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\") pod \"208a62c2-2057-4e6a-9810-b9562441e759\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " Nov 01 05:58:23.381412 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.381400 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\" (UniqueName: \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\") pod \"208a62c2-2057-4e6a-9810-b9562441e759\" (UID: \"208a62c2-2057-4e6a-9810-b9562441e759\") " Nov 01 05:58:23.381772 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.381740 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca" pod "208a62c2-2057-4e6a-9810-b9562441e759" (UID: "208a62c2-2057-4e6a-9810-b9562441e759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:58:23.383451 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.383431 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca" pod "208a62c2-2057-4e6a-9810-b9562441e759" (UID: "208a62c2-2057-4e6a-9810-b9562441e759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:58:23.383632 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.383608 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27" pod "208a62c2-2057-4e6a-9810-b9562441e759" (UID: "208a62c2-2057-4e6a-9810-b9562441e759"). InnerVolumeSpecName "kube-api-access-hht27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:58:23.483431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.483399 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:23.483431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.483432 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a62c2-2057-4e6a-9810-b9562441e759-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:23.483675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:23.483443 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hht27\" (UniqueName: \"kubernetes.io/projected/208a62c2-2057-4e6a-9810-b9562441e759-kube-api-access-hht27\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:24.208927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:24.208896 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" Nov 01 05:58:24.209103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:24.208935 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp" event={"ID":"208a62c2-2057-4e6a-9810-b9562441e759","Type":"ContainerDied","Data":"d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436"} Nov 01 05:58:24.209103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:24.208983 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d108a57232db3b7b62edf4ad2237ba6c1c107e943770c318899455b78ce81436" Nov 01 05:58:26.948484 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:26.948436 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:58:31.047541 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:31.047508 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c"] Nov 01 05:58:31.098122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:31.098084 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b10d23-xrb4c"] Nov 01 05:58:32.211558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.211515 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2"] Nov 01 05:58:32.212279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.212246 4425 cpu_manager.go:414] "containerMap: removing container" podUID="208a62c2-2057-4e6a-9810-b9562441e759" containerName="revoke-keys" Nov 01 05:58:32.212344 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.212281 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a62c2-2057-4e6a-9810-b9562441e759" containerName="revoke-keys" Nov 01 05:58:32.212388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.212379 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a62c2-2057-4e6a-9810-b9562441e759" containerName="revoke-keys" Nov 01 05:58:32.216326 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.216306 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.272692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.272662 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2"] Nov 01 05:58:32.310101 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.310064 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdr99\" (UniqueName: \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.310337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.310125 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.310337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.310271 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.319301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.319244 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:58:32.319301 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.319246 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:58:32.319487 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.319419 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:58:32.319530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.319503 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:58:32.412100 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.412061 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.412327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.412289 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdr99\" (UniqueName: \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.412388 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.412347 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.413019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.412989 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.414722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.414699 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.474436 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.474353 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdr99\" (UniqueName: \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\") pod \"maas-api-revoke-sub-b2071016-gf2m2\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.530400 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.530367 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:32.911341 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.911316 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2"] Nov 01 05:58:32.911647 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:32.911625 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e706358_9563_482d_837d_0d067ded516a.slice/crio-c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99 WatchSource:0}: Error finding container c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99: Status 404 returned error can't find the container with id c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99 Nov 01 05:58:32.981838 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:32.981802 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9451806a-d97b-4e12-94f4-0ef7a06a8d86" path="/var/lib/kubelet/pods/9451806a-d97b-4e12-94f4-0ef7a06a8d86/volumes" Nov 01 05:58:33.204621 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:33.204530 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 05:58:33.271477 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:33.271447 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" event={"ID":"0e706358-9563-482d-837d-0d067ded516a","Type":"ContainerStarted","Data":"74cd2823d7af6b8a24b8a870521578e252d02971515872d9fa69fe6cb752ba0a"} Nov 01 05:58:33.271872 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:33.271502 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" event={"ID":"0e706358-9563-482d-837d-0d067ded516a","Type":"ContainerStarted","Data":"c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99"} Nov 01 05:58:33.439965 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:33.439908 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" podStartSLOduration=1.439889333 podStartE2EDuration="1.439889333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:33.438755448 +0000 UTC m=+3328.579344367" watchObservedRunningTime="2026-11-01 05:58:33.439889333 +0000 UTC m=+3328.580478263" Nov 01 05:58:34.280552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.280452 4425 generic.go:356] "Generic (PLEG): container finished" podID="0e706358-9563-482d-837d-0d067ded516a" containerID="74cd2823d7af6b8a24b8a870521578e252d02971515872d9fa69fe6cb752ba0a" exitCode=0 Nov 01 05:58:34.280552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.280536 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" event={"ID":"0e706358-9563-482d-837d-0d067ded516a","Type":"ContainerDied","Data":"74cd2823d7af6b8a24b8a870521578e252d02971515872d9fa69fe6cb752ba0a"} Nov 01 05:58:34.508503 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.508469 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:58:34.512874 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.512851 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.570647 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.570615 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:58:34.641926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.641889 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.642122 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.642031 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ms4r\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.642224 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.642204 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.642332 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.642239 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.743930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.743893 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.743930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.743938 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.744218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.743993 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.744218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.744060 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ms4r\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.744803 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.744777 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.744901 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.744836 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.747146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.747123 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.806072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.806041 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ms4r\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:34.827427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:34.827346 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:35.218845 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.218818 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:58:35.219918 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:35.219886 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59173380_1a68_42cf_95e5_d6301ad32e39.slice/crio-ed4045d57f2a68e7947c44fb4106d83e72edc3db5cab18f930ba3a92f90fff3e WatchSource:0}: Error finding container ed4045d57f2a68e7947c44fb4106d83e72edc3db5cab18f930ba3a92f90fff3e: Status 404 returned error can't find the container with id ed4045d57f2a68e7947c44fb4106d83e72edc3db5cab18f930ba3a92f90fff3e Nov 01 05:58:35.289572 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.289542 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" event={"ID":"59173380-1a68-42cf-95e5-d6301ad32e39","Type":"ContainerStarted","Data":"ed4045d57f2a68e7947c44fb4106d83e72edc3db5cab18f930ba3a92f90fff3e"} Nov 01 05:58:35.455241 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.455210 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:35.564879 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.564848 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\" (UniqueName: \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\") pod \"0e706358-9563-482d-837d-0d067ded516a\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " Nov 01 05:58:35.565118 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.565042 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\") pod \"0e706358-9563-482d-837d-0d067ded516a\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " Nov 01 05:58:35.565196 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.565170 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\") pod \"0e706358-9563-482d-837d-0d067ded516a\" (UID: \"0e706358-9563-482d-837d-0d067ded516a\") " Nov 01 05:58:35.566429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.566398 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca" pod "0e706358-9563-482d-837d-0d067ded516a" (UID: "0e706358-9563-482d-837d-0d067ded516a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:58:35.568642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.568509 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca" pod "0e706358-9563-482d-837d-0d067ded516a" (UID: "0e706358-9563-482d-837d-0d067ded516a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:58:35.568642 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.568598 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99" pod "0e706358-9563-482d-837d-0d067ded516a" (UID: "0e706358-9563-482d-837d-0d067ded516a"). InnerVolumeSpecName "kube-api-access-rdr99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:58:35.681926 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.675679 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e706358-9563-482d-837d-0d067ded516a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:35.682209 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.682185 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdr99\" (UniqueName: \"kubernetes.io/projected/0e706358-9563-482d-837d-0d067ded516a-kube-api-access-rdr99\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:35.682349 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:35.682335 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e706358-9563-482d-837d-0d067ded516a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:36.298226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.298191 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" Nov 01 05:58:36.298226 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.298210 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2" event={"ID":"0e706358-9563-482d-837d-0d067ded516a","Type":"ContainerDied","Data":"c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99"} Nov 01 05:58:36.298806 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.298303 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8e3abd17e39f30c570089a4e20fe2086b9e2989e6da5859a384a84e50429a99" Nov 01 05:58:36.300042 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.300015 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" event={"ID":"59173380-1a68-42cf-95e5-d6301ad32e39","Type":"ContainerStarted","Data":"db0fd0acc2a03c2033fc922da3aa17be561280cbd15ed4f506fcce9345a7a156"} Nov 01 05:58:36.300235 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.300220 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:58:36.470217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:36.470163 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" podStartSLOduration=2.470147286 podStartE2EDuration="2.470147286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:36.466577266 +0000 UTC m=+3331.607166208" watchObservedRunningTime="2026-11-01 05:58:36.470147286 +0000 UTC m=+3331.610736239" Nov 01 05:58:41.048166 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:41.048126 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw"] Nov 01 05:58:41.098454 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:41.098407 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-121edbdf-kwrbw"] Nov 01 05:58:42.975163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:42.975129 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4474e0cb-34c2-4a07-b3ca-96b7313b9df6" path="/var/lib/kubelet/pods/4474e0cb-34c2-4a07-b3ca-96b7313b9df6/volumes" Nov 01 05:58:43.992198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:43.992164 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr"] Nov 01 05:58:43.992930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:43.992911 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0e706358-9563-482d-837d-0d067ded516a" containerName="revoke-keys" Nov 01 05:58:43.992930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:43.992929 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e706358-9563-482d-837d-0d067ded516a" containerName="revoke-keys" Nov 01 05:58:43.993089 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:43.993068 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e706358-9563-482d-837d-0d067ded516a" containerName="revoke-keys" Nov 01 05:58:43.998866 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:43.998836 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.055012 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.054975 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr"] Nov 01 05:58:44.098354 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.098315 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.098535 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.098403 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.098708 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.098678 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6bx\" (UniqueName: \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.100082 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.100056 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:58:44.100238 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.100175 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:58:44.117859 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.117827 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:58:44.118003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.117870 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:58:44.200046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.200004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.200269 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.200236 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6bx\" (UniqueName: \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.200373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.200358 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.201158 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.201130 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.203046 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.203014 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.262373 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.262289 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6bx\" (UniqueName: \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\") pod \"maas-api-revoke-sub-27cb0565-whvwr\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.320110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.320067 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:44.720522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:44.720499 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr"] Nov 01 05:58:44.721484 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:44.721456 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0300dffd_9474_48e3_9e97_46cde7d46c99.slice/crio-c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d WatchSource:0}: Error finding container c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d: Status 404 returned error can't find the container with id c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d Nov 01 05:58:45.387365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:45.387326 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" event={"ID":"0300dffd-9474-48e3-9e97-46cde7d46c99","Type":"ContainerStarted","Data":"d61b2c3674ab5d00585eceabeffadab0910e64b5c03b2c91095fc5747a44622a"} Nov 01 05:58:45.387779 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:45.387394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" event={"ID":"0300dffd-9474-48e3-9e97-46cde7d46c99","Type":"ContainerStarted","Data":"c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d"} Nov 01 05:58:45.506739 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:45.506680 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" podStartSLOduration=2.5066607960000002 podStartE2EDuration="2.506660796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:45.503740757 +0000 UTC m=+3340.644329677" watchObservedRunningTime="2026-11-01 05:58:45.506660796 +0000 UTC m=+3340.647249717" Nov 01 05:58:46.396529 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:46.396498 4425 generic.go:356] "Generic (PLEG): container finished" podID="0300dffd-9474-48e3-9e97-46cde7d46c99" containerID="d61b2c3674ab5d00585eceabeffadab0910e64b5c03b2c91095fc5747a44622a" exitCode=0 Nov 01 05:58:46.396943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:46.396595 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" event={"ID":"0300dffd-9474-48e3-9e97-46cde7d46c99","Type":"ContainerDied","Data":"d61b2c3674ab5d00585eceabeffadab0910e64b5c03b2c91095fc5747a44622a"} Nov 01 05:58:47.311297 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.311238 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:58:47.568092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.568014 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:47.648017 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.647978 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\") pod \"0300dffd-9474-48e3-9e97-46cde7d46c99\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " Nov 01 05:58:47.648237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.648090 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\") pod \"0300dffd-9474-48e3-9e97-46cde7d46c99\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " Nov 01 05:58:47.648237 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.648141 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\" (UniqueName: \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\") pod \"0300dffd-9474-48e3-9e97-46cde7d46c99\" (UID: \"0300dffd-9474-48e3-9e97-46cde7d46c99\") " Nov 01 05:58:47.648522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.648494 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca" pod "0300dffd-9474-48e3-9e97-46cde7d46c99" (UID: "0300dffd-9474-48e3-9e97-46cde7d46c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:58:47.648941 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.648908 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:47.650372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.650336 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca" pod "0300dffd-9474-48e3-9e97-46cde7d46c99" (UID: "0300dffd-9474-48e3-9e97-46cde7d46c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:58:47.650546 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.650521 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx" pod "0300dffd-9474-48e3-9e97-46cde7d46c99" (UID: "0300dffd-9474-48e3-9e97-46cde7d46c99"). InnerVolumeSpecName "kube-api-access-xl6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:58:47.750112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.750074 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6bx\" (UniqueName: \"kubernetes.io/projected/0300dffd-9474-48e3-9e97-46cde7d46c99-kube-api-access-xl6bx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:47.750112 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:47.750111 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0300dffd-9474-48e3-9e97-46cde7d46c99-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:58:48.412304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:48.412245 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" Nov 01 05:58:48.412304 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:48.412278 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr" event={"ID":"0300dffd-9474-48e3-9e97-46cde7d46c99","Type":"ContainerDied","Data":"c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d"} Nov 01 05:58:48.412526 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:48.412338 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27a5717bc54cb3c422eb68dbe17b85ae8ce8558da6ce5bad07c4385525c894d" Nov 01 05:58:56.948926 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:56.948888 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:58:57.308955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.308869 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:58:57.396375 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.396340 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:58:57.397663 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.397630 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:58:57.427473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.427432 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:58:57.428080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.428067 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0300dffd-9474-48e3-9e97-46cde7d46c99" containerName="revoke-keys" Nov 01 05:58:57.428133 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.428082 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0300dffd-9474-48e3-9e97-46cde7d46c99" containerName="revoke-keys" Nov 01 05:58:57.428230 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.428220 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0300dffd-9474-48e3-9e97-46cde7d46c99" containerName="revoke-keys" Nov 01 05:58:57.433011 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.432993 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.508320 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.508284 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:58:57.580929 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.580892 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw8h\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.581148 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.581113 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.581222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.581153 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.581222 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.581180 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.682728 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.682689 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.682927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.682737 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.682927 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.682862 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.683019 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.682982 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw8h\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.683530 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.683505 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.683675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.683655 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.685630 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.685607 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.745427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.745389 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw8h\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:57.747339 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:57.747315 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:58.131695 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:58.131666 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:58:58.132062 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:58:58.132011 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a97562_7d29_4034_92ed_b7f0bc7c46c2.slice/crio-0d21df3ffa9823a2e946d754605b95d67ffe44eed1fb256eeafda290e6ce3e45 WatchSource:0}: Error finding container 0d21df3ffa9823a2e946d754605b95d67ffe44eed1fb256eeafda290e6ce3e45: Status 404 returned error can't find the container with id 0d21df3ffa9823a2e946d754605b95d67ffe44eed1fb256eeafda290e6ce3e45 Nov 01 05:58:58.485887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:58.485784 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" event={"ID":"42a97562-7d29-4034-92ed-b7f0bc7c46c2","Type":"ContainerStarted","Data":"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2"} Nov 01 05:58:58.485887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:58.485854 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" event={"ID":"42a97562-7d29-4034-92ed-b7f0bc7c46c2","Type":"ContainerStarted","Data":"0d21df3ffa9823a2e946d754605b95d67ffe44eed1fb256eeafda290e6ce3e45"} Nov 01 05:58:58.486128 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:58.485998 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:58:58.494945 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:58.494913 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:58.494945 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:58.494955 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:58.495206 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:58.495026 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials podName:42a97562-7d29-4034-92ed-b7f0bc7c46c2 nodeName:}" failed. No retries permitted until 2026-11-01 05:58:58.995004052 +0000 UTC m=+3354.135592964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2") : secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:58.605348 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:58.605289 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" podStartSLOduration=1.605272823 podStartE2EDuration="1.605272823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:58.603086213 +0000 UTC m=+3353.743675153" watchObservedRunningTime="2026-11-01 05:58:58.605272823 +0000 UTC m=+3353.745861789" Nov 01 05:58:59.004597 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:59.004560 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:59.004597 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:59.004599 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:59.004881 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:58:59.004677 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials podName:42a97562-7d29-4034-92ed-b7f0bc7c46c2 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:00.004657199 +0000 UTC m=+3355.145246125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2") : secret "agc-cred-0a08dbd11a06" not found Nov 01 05:58:59.459167 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.458297 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf"] Nov 01 05:58:59.474678 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.474646 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.512585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.512551 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkkbw\" (UniqueName: \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.513212 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.513186 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.513441 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.513407 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.515378 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.515343 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf"] Nov 01 05:58:59.528077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.528050 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:58:59.528216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.528052 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:58:59.528216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.528102 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:58:59.528337 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.528223 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:58:59.615121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.615087 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.615121 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.615132 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.615405 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.615208 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkkbw\" (UniqueName: \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.615903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.615833 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.617742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.617718 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.676839 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.676810 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkkbw\" (UniqueName: \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\") pod \"maas-api-revoke-sub-3cbf8dbb-pdsrf\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:58:59.794656 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:58:59.794574 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:59:00.011693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:00.011657 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:59:00.020954 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:00.020924 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:00.020954 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:00.020957 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:00.021149 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:00.021014 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials podName:42a97562-7d29-4034-92ed-b7f0bc7c46c2 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:02.02099995 +0000 UTC m=+3357.161588847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2") : secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:00.175542 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:00.175515 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf"] Nov 01 05:59:00.176454 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:59:00.176423 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64b4217_8e8e_416d_9570_d8311c936a1f.slice/crio-c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e WatchSource:0}: Error finding container c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e: Status 404 returned error can't find the container with id c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e Nov 01 05:59:00.504022 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:00.503986 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" event={"ID":"c64b4217-8e8e-416d-9570-d8311c936a1f","Type":"ContainerStarted","Data":"6e46b357a901f0589597ae0bd9dfdd5a667c630d960155ae7eee3a62f75acb58"} Nov 01 05:59:00.504512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:00.504049 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" event={"ID":"c64b4217-8e8e-416d-9570-d8311c936a1f","Type":"ContainerStarted","Data":"c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e"} Nov 01 05:59:00.624040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:00.623987 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" podStartSLOduration=1.623972977 podStartE2EDuration="1.623972977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:00.621743956 +0000 UTC m=+3355.762332875" watchObservedRunningTime="2026-11-01 05:59:00.623972977 +0000 UTC m=+3355.764561895" Nov 01 05:59:01.512537 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:01.512500 4425 generic.go:356] "Generic (PLEG): container finished" podID="c64b4217-8e8e-416d-9570-d8311c936a1f" containerID="6e46b357a901f0589597ae0bd9dfdd5a667c630d960155ae7eee3a62f75acb58" exitCode=0 Nov 01 05:59:01.512980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:01.512586 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" event={"ID":"c64b4217-8e8e-416d-9570-d8311c936a1f","Type":"ContainerDied","Data":"6e46b357a901f0589597ae0bd9dfdd5a667c630d960155ae7eee3a62f75acb58"} Nov 01 05:59:02.046567 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:02.046526 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:02.046567 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:02.046568 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:02.046825 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:02.046620 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials podName:42a97562-7d29-4034-92ed-b7f0bc7c46c2 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:06.046606615 +0000 UTC m=+3361.187195525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2") : secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:02.397557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.397516 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" containerID="cri-o://db0fd0acc2a03c2033fc922da3aa17be561280cbd15ed4f506fcce9345a7a156" gracePeriod=25 Nov 01 05:59:02.524469 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.524436 4425 generic.go:356] "Generic (PLEG): container finished" podID="59173380-1a68-42cf-95e5-d6301ad32e39" containerID="db0fd0acc2a03c2033fc922da3aa17be561280cbd15ed4f506fcce9345a7a156" exitCode=0 Nov 01 05:59:02.524870 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.524509 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" event={"ID":"59173380-1a68-42cf-95e5-d6301ad32e39","Type":"ContainerDied","Data":"db0fd0acc2a03c2033fc922da3aa17be561280cbd15ed4f506fcce9345a7a156"} Nov 01 05:59:02.698053 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.698022 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:59:02.733172 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.733145 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:59:02.754623 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754589 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\" (UniqueName: \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\") pod \"c64b4217-8e8e-416d-9570-d8311c936a1f\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " Nov 01 05:59:02.754793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754681 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\") pod \"59173380-1a68-42cf-95e5-d6301ad32e39\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " Nov 01 05:59:02.754793 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754759 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\") pod \"59173380-1a68-42cf-95e5-d6301ad32e39\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " Nov 01 05:59:02.754925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754826 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\") pod \"59173380-1a68-42cf-95e5-d6301ad32e39\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " Nov 01 05:59:02.754925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754868 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\") pod \"59173380-1a68-42cf-95e5-d6301ad32e39\" (UID: \"59173380-1a68-42cf-95e5-d6301ad32e39\") " Nov 01 05:59:02.755045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754966 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\") pod \"c64b4217-8e8e-416d-9570-d8311c936a1f\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " Nov 01 05:59:02.755045 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.754998 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\") pod \"c64b4217-8e8e-416d-9570-d8311c936a1f\" (UID: \"c64b4217-8e8e-416d-9570-d8311c936a1f\") " Nov 01 05:59:02.755716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.755451 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume" pod "59173380-1a68-42cf-95e5-d6301ad32e39" (UID: "59173380-1a68-42cf-95e5-d6301ad32e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:02.755716 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.755687 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.756135 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.756101 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume" pod "59173380-1a68-42cf-95e5-d6301ad32e39" (UID: "59173380-1a68-42cf-95e5-d6301ad32e39"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:02.756211 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.756146 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca" pod "c64b4217-8e8e-416d-9570-d8311c936a1f" (UID: "c64b4217-8e8e-416d-9570-d8311c936a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:02.757401 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.757381 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r" pod "59173380-1a68-42cf-95e5-d6301ad32e39" (UID: "59173380-1a68-42cf-95e5-d6301ad32e39"). InnerVolumeSpecName "kube-api-access-4ms4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:02.757574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.757554 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials" pod "59173380-1a68-42cf-95e5-d6301ad32e39" (UID: "59173380-1a68-42cf-95e5-d6301ad32e39"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:02.757634 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.757564 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw" pod "c64b4217-8e8e-416d-9570-d8311c936a1f" (UID: "c64b4217-8e8e-416d-9570-d8311c936a1f"). InnerVolumeSpecName "kube-api-access-lkkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:02.757817 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.757792 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca" pod "c64b4217-8e8e-416d-9570-d8311c936a1f" (UID: "c64b4217-8e8e-416d-9570-d8311c936a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:59:02.857142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857103 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkkbw\" (UniqueName: \"kubernetes.io/projected/c64b4217-8e8e-416d-9570-d8311c936a1f-kube-api-access-lkkbw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.857142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857136 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ms4r\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-kube-api-access-4ms4r\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.857142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857146 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59173380-1a68-42cf-95e5-d6301ad32e39-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.857522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857156 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59173380-1a68-42cf-95e5-d6301ad32e39-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.857522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857167 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:02.857522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:02.857175 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64b4217-8e8e-416d-9570-d8311c936a1f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:03.319889 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.319858 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq"] Nov 01 05:59:03.320611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320592 4425 cpu_manager.go:414] "containerMap: removing container" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" Nov 01 05:59:03.320611 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320611 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" Nov 01 05:59:03.320792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320647 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c64b4217-8e8e-416d-9570-d8311c936a1f" containerName="revoke-keys" Nov 01 05:59:03.320792 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320655 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64b4217-8e8e-416d-9570-d8311c936a1f" containerName="revoke-keys" Nov 01 05:59:03.320903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320856 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" containerName="payload-processing" Nov 01 05:59:03.320903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.320873 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64b4217-8e8e-416d-9570-d8311c936a1f" containerName="revoke-keys" Nov 01 05:59:03.324875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.324851 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.363930 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.363893 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.364092 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.363938 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzk9f\" (UniqueName: \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.364138 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.364091 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.405748 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.405715 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq"] Nov 01 05:59:03.465508 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.465469 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.465687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.465519 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzk9f\" (UniqueName: \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.465687 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.465633 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.466377 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.466355 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.468137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.468118 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.529150 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.529115 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzk9f\" (UniqueName: \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\") pod \"maas-api-revoke-sub-478aff25-cv6rq\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.544582 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.544553 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" Nov 01 05:59:03.544900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.544694 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf" event={"ID":"c64b4217-8e8e-416d-9570-d8311c936a1f","Type":"ContainerDied","Data":"c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e"} Nov 01 05:59:03.544900 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.544775 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c99204ad874e33ba41a63103ca34e77725839164965640ff75ae8773f8513c1e" Nov 01 05:59:03.547442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.547399 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" event={"ID":"59173380-1a68-42cf-95e5-d6301ad32e39","Type":"ContainerDied","Data":"ed4045d57f2a68e7947c44fb4106d83e72edc3db5cab18f930ba3a92f90fff3e"} Nov 01 05:59:03.547442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.547438 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z" Nov 01 05:59:03.547638 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.547465 4425 scope.go:122] "RemoveContainer" containerID="db0fd0acc2a03c2033fc922da3aa17be561280cbd15ed4f506fcce9345a7a156" Nov 01 05:59:03.639077 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.638987 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:03.716198 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.716163 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:59:03.721852 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:03.721817 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-76b72nw9z"] Nov 01 05:59:04.021849 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:04.021821 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq"] Nov 01 05:59:04.022004 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:59:04.021846 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5560761c_148d_4c7b_ad39_8f96eddcbd3d.slice/crio-027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503 WatchSource:0}: Error finding container 027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503: Status 404 returned error can't find the container with id 027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503 Nov 01 05:59:04.556857 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:04.556821 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" event={"ID":"5560761c-148d-4c7b-ad39-8f96eddcbd3d","Type":"ContainerStarted","Data":"aa0787e8a8b14dec69b261f195bf95b0e245803c7e9dd3c45ac96a0a4c76e096"} Nov 01 05:59:04.557329 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:04.556892 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" event={"ID":"5560761c-148d-4c7b-ad39-8f96eddcbd3d","Type":"ContainerStarted","Data":"027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503"} Nov 01 05:59:04.676143 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:04.676022 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" podStartSLOduration=1.6760085 podStartE2EDuration="1.6760085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:04.673030467 +0000 UTC m=+3359.813619406" watchObservedRunningTime="2026-11-01 05:59:04.6760085 +0000 UTC m=+3359.816597419" Nov 01 05:59:04.975556 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:04.975523 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59173380-1a68-42cf-95e5-d6301ad32e39" path="/var/lib/kubelet/pods/59173380-1a68-42cf-95e5-d6301ad32e39/volumes" Nov 01 05:59:05.504813 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.504764 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" podUID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" containerName="payload-processing" containerID="cri-o://2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2" gracePeriod=25 Nov 01 05:59:05.567490 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.567457 4425 generic.go:356] "Generic (PLEG): container finished" podID="5560761c-148d-4c7b-ad39-8f96eddcbd3d" containerID="aa0787e8a8b14dec69b261f195bf95b0e245803c7e9dd3c45ac96a0a4c76e096" exitCode=0 Nov 01 05:59:05.567972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.567550 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" event={"ID":"5560761c-148d-4c7b-ad39-8f96eddcbd3d","Type":"ContainerDied","Data":"aa0787e8a8b14dec69b261f195bf95b0e245803c7e9dd3c45ac96a0a4c76e096"} Nov 01 05:59:05.818595 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.818566 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:59:05.899692 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.899652 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\") pod \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " Nov 01 05:59:05.899894 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.899721 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\") pod \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " Nov 01 05:59:05.899961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.899902 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\") pod \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " Nov 01 05:59:05.899961 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.899942 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\") pod \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\" (UID: \"42a97562-7d29-4034-92ed-b7f0bc7c46c2\") " Nov 01 05:59:05.900239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.900210 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume" pod "42a97562-7d29-4034-92ed-b7f0bc7c46c2" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:05.900381 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.900337 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume" pod "42a97562-7d29-4034-92ed-b7f0bc7c46c2" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:05.900785 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.900763 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:05.900835 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.900793 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42a97562-7d29-4034-92ed-b7f0bc7c46c2-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:05.902106 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.902082 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials" pod "42a97562-7d29-4034-92ed-b7f0bc7c46c2" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:05.902199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:05.902177 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h" pod "42a97562-7d29-4034-92ed-b7f0bc7c46c2" (UID: "42a97562-7d29-4034-92ed-b7f0bc7c46c2"). InnerVolumeSpecName "kube-api-access-vsw8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:06.002479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.002446 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:06.002479 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.002476 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsw8h\" (UniqueName: \"kubernetes.io/projected/42a97562-7d29-4034-92ed-b7f0bc7c46c2-kube-api-access-vsw8h\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:06.576538 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.576504 4425 generic.go:356] "Generic (PLEG): container finished" podID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" containerID="2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2" exitCode=0 Nov 01 05:59:06.576978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.576576 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" Nov 01 05:59:06.576978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.576587 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" event={"ID":"42a97562-7d29-4034-92ed-b7f0bc7c46c2","Type":"ContainerDied","Data":"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2"} Nov 01 05:59:06.576978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.576669 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl" event={"ID":"42a97562-7d29-4034-92ed-b7f0bc7c46c2","Type":"ContainerDied","Data":"0d21df3ffa9823a2e946d754605b95d67ffe44eed1fb256eeafda290e6ce3e45"} Nov 01 05:59:06.576978 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.576693 4425 scope.go:122] "RemoveContainer" containerID="2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2" Nov 01 05:59:06.588562 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.588533 4425 scope.go:122] "RemoveContainer" containerID="2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2" Nov 01 05:59:06.588917 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:06.588888 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2\": container with ID starting with 2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2 not found: ID does not exist" containerID="2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2" Nov 01 05:59:06.589009 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.588931 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2"} err="failed to get container status \"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2\": rpc error: code = NotFound desc = could not find container \"2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2\": container with ID starting with 2365bc5ad5553259814217f7ee2ee1b71b2dab55288858271bfe510f17a8e9c2 not found: ID does not exist" Nov 01 05:59:06.742474 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.742443 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:06.747379 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.747354 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:59:06.750773 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.750748 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-6bdclmmsl"] Nov 01 05:59:06.811694 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.811661 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\") pod \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " Nov 01 05:59:06.811887 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.811828 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\") pod \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " Nov 01 05:59:06.811942 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.811895 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\" (UniqueName: \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\") pod \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\" (UID: \"5560761c-148d-4c7b-ad39-8f96eddcbd3d\") " Nov 01 05:59:06.812210 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.812187 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca" pod "5560761c-148d-4c7b-ad39-8f96eddcbd3d" (UID: "5560761c-148d-4c7b-ad39-8f96eddcbd3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:06.812442 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.812424 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:06.814103 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.814078 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca" pod "5560761c-148d-4c7b-ad39-8f96eddcbd3d" (UID: "5560761c-148d-4c7b-ad39-8f96eddcbd3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:59:06.814216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.814200 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f" pod "5560761c-148d-4c7b-ad39-8f96eddcbd3d" (UID: "5560761c-148d-4c7b-ad39-8f96eddcbd3d"). InnerVolumeSpecName "kube-api-access-vzk9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:06.913552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.913459 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzk9f\" (UniqueName: \"kubernetes.io/projected/5560761c-148d-4c7b-ad39-8f96eddcbd3d-kube-api-access-vzk9f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:06.913552 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.913490 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5560761c-148d-4c7b-ad39-8f96eddcbd3d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:06.975124 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:06.975087 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" path="/var/lib/kubelet/pods/42a97562-7d29-4034-92ed-b7f0bc7c46c2/volumes" Nov 01 05:59:07.586429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:07.586397 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" Nov 01 05:59:07.586429 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:07.586421 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq" event={"ID":"5560761c-148d-4c7b-ad39-8f96eddcbd3d","Type":"ContainerDied","Data":"027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503"} Nov 01 05:59:07.586883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:07.586469 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027edfafe7d88ebd5f7de5d4fffd0fb011a1d048ca6afd5d68aa9e46cb454503" Nov 01 05:59:07.718980 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:07.718953 4425 scope.go:122] "RemoveContainer" containerID="8cdc528b62add34c9ff0bb0bd97e9536b18651cc9ffeaf5b1c2edd412dd89d75" Nov 01 05:59:07.729742 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:07.729717 4425 scope.go:122] "RemoveContainer" containerID="f4ff5fb0ffec167316848aa3668e78aa3a830ef321f6cda57efc977753f2b32c" Nov 01 05:59:13.710280 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.697171 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb"] Nov 01 05:59:13.712033 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712000 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5560761c-148d-4c7b-ad39-8f96eddcbd3d" containerName="revoke-keys" Nov 01 05:59:13.712267 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712231 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5560761c-148d-4c7b-ad39-8f96eddcbd3d" containerName="revoke-keys" Nov 01 05:59:13.712419 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712403 4425 cpu_manager.go:414] "containerMap: removing container" podUID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" containerName="payload-processing" Nov 01 05:59:13.712574 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712558 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" containerName="payload-processing" Nov 01 05:59:13.712875 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712858 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5560761c-148d-4c7b-ad39-8f96eddcbd3d" containerName="revoke-keys" Nov 01 05:59:13.713004 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.712992 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a97562-7d29-4034-92ed-b7f0bc7c46c2" containerName="payload-processing" Nov 01 05:59:13.729217 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.729180 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.740216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.740181 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb"] Nov 01 05:59:13.785520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.785489 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 05:59:13.785520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.785489 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 05:59:13.785520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.785490 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 05:59:13.785520 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.785533 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 05:59:13.804168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.804130 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kjc\" (UniqueName: \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.804168 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.804175 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.804447 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.804425 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.915959 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.915924 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kjc\" (UniqueName: \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.916142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.915973 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.916142 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.916122 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.916883 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.916854 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.919163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.919135 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:13.978327 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:13.978205 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kjc\" (UniqueName: \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\") pod \"maas-api-revoke-sub-c4aabf48-k76pb\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:14.048722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:14.048681 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121047 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121098 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121109 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121118 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121139 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:14.121389 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.121210 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:14.621186117 +0000 UTC m=+3369.761775032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:14.429619 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:14.429589 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb"] Nov 01 05:59:14.430190 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:59:14.430167 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36397df4_3586_4802_a7aa_88d303e3927e.slice/crio-6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567 WatchSource:0}: Error finding container 6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567: Status 404 returned error can't find the container with id 6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567 Nov 01 05:59:14.628307 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628273 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:14.628307 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628302 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:14.628307 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628313 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:14.628561 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628322 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:14.628561 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628336 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:14.628561 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:14.628393 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:15.628379278 +0000 UTC m=+3370.768968196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:14.639085 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:14.639054 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" event={"ID":"36397df4-3586-4802-a7aa-88d303e3927e","Type":"ContainerStarted","Data":"6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567"} Nov 01 05:59:15.108588 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.108547 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj"] Nov 01 05:59:15.112862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.112843 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.170764 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.170723 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj"] Nov 01 05:59:15.235273 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.235213 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.235463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.235367 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.235463 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.235408 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pg4\" (UniqueName: \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.336324 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.336293 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.336472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.336362 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.336472 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.336393 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pg4\" (UniqueName: \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.336972 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.336951 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.338690 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.338663 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.382512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.382436 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:59:15.398501 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.398467 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pg4\" (UniqueName: \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\") pod \"maas-api-revoke-sub-ebfca4d0-gvjkj\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.426925 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.426901 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:15.489218 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.488962 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:59:15.640636 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640567 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:15.640636 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640591 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:15.640636 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640599 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:15.640636 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640606 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:15.640636 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640619 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:15.640912 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:15.640672 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:17.64065838 +0000 UTC m=+3372.781247291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:15.647907 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.647875 4425 generic.go:356] "Generic (PLEG): container finished" podID="36397df4-3586-4802-a7aa-88d303e3927e" containerID="09182654c540a61c598a353594841efdbcb59d80f8b4df45561d7f1820bc9515" exitCode=0 Nov 01 05:59:15.648043 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.647964 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" event={"ID":"36397df4-3586-4802-a7aa-88d303e3927e","Type":"ContainerDied","Data":"09182654c540a61c598a353594841efdbcb59d80f8b4df45561d7f1820bc9515"} Nov 01 05:59:15.831216 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:15.831190 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj"] Nov 01 05:59:16.657558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:16.657516 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" event={"ID":"3f21dc0b-5ee6-4300-88c4-906e6ba44047","Type":"ContainerStarted","Data":"173b809eaa537c5c5590dfe03aaa5bff981c7600b4320cb3ec70c521b35a0c88"} Nov 01 05:59:16.658029 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:16.657586 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" event={"ID":"3f21dc0b-5ee6-4300-88c4-906e6ba44047","Type":"ContainerStarted","Data":"5f2a661a32bf6feb5f062b7e95a02b7043cb6f4be3e49950b0b7079e024adb4a"} Nov 01 05:59:16.779003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:16.778944 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" podStartSLOduration=1.7789255320000001 podStartE2EDuration="1.778925532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:16.775209459 +0000 UTC m=+3371.915798396" watchObservedRunningTime="2026-11-01 05:59:16.778925532 +0000 UTC m=+3371.919514452" Nov 01 05:59:16.952453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:16.952388 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:17.059991 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.059959 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\") pod \"36397df4-3586-4802-a7aa-88d303e3927e\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " Nov 01 05:59:17.060177 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.060144 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\" (UniqueName: \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\") pod \"36397df4-3586-4802-a7aa-88d303e3927e\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " Nov 01 05:59:17.060244 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.060200 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\") pod \"36397df4-3586-4802-a7aa-88d303e3927e\" (UID: \"36397df4-3586-4802-a7aa-88d303e3927e\") " Nov 01 05:59:17.060403 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.060379 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca" pod "36397df4-3586-4802-a7aa-88d303e3927e" (UID: "36397df4-3586-4802-a7aa-88d303e3927e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:17.060641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.060627 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:17.062516 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.062491 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc" pod "36397df4-3586-4802-a7aa-88d303e3927e" (UID: "36397df4-3586-4802-a7aa-88d303e3927e"). InnerVolumeSpecName "kube-api-access-68kjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:17.062616 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.062598 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca" pod "36397df4-3586-4802-a7aa-88d303e3927e" (UID: "36397df4-3586-4802-a7aa-88d303e3927e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:59:17.161743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.161711 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kjc\" (UniqueName: \"kubernetes.io/projected/36397df4-3586-4802-a7aa-88d303e3927e-kube-api-access-68kjc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:17.161743 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.161741 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36397df4-3586-4802-a7aa-88d303e3927e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:17.667987 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.667948 4425 generic.go:356] "Generic (PLEG): container finished" podID="3f21dc0b-5ee6-4300-88c4-906e6ba44047" containerID="173b809eaa537c5c5590dfe03aaa5bff981c7600b4320cb3ec70c521b35a0c88" exitCode=0 Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.668116 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" event={"ID":"3f21dc0b-5ee6-4300-88c4-906e6ba44047","Type":"ContainerDied","Data":"173b809eaa537c5c5590dfe03aaa5bff981c7600b4320cb3ec70c521b35a0c88"} Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668359 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668380 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668393 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668404 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668420 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:17.668522 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:17.668485 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:21.66846816 +0000 UTC m=+3376.809057059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:17.669857 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.669838 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" Nov 01 05:59:17.670003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.669875 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb" event={"ID":"36397df4-3586-4802-a7aa-88d303e3927e","Type":"ContainerDied","Data":"6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567"} Nov 01 05:59:17.670003 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:17.669944 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d77d378bec2329d77b03405285cf770d2d604ba3f3496e89f70d5401bae9567" Nov 01 05:59:18.832330 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.832306 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:18.985398 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.985299 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\") pod \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " Nov 01 05:59:18.985557 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.985440 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\") pod \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " Nov 01 05:59:18.985608 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.985561 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\" (UniqueName: \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\") pod \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\" (UID: \"3f21dc0b-5ee6-4300-88c4-906e6ba44047\") " Nov 01 05:59:18.985721 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.985691 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca" pod "3f21dc0b-5ee6-4300-88c4-906e6ba44047" (UID: "3f21dc0b-5ee6-4300-88c4-906e6ba44047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:18.986402 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.986375 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:18.987871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.987843 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca" pod "3f21dc0b-5ee6-4300-88c4-906e6ba44047" (UID: "3f21dc0b-5ee6-4300-88c4-906e6ba44047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 05:59:18.988098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:18.988073 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4" pod "3f21dc0b-5ee6-4300-88c4-906e6ba44047" (UID: "3f21dc0b-5ee6-4300-88c4-906e6ba44047"). InnerVolumeSpecName "kube-api-access-l6pg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:19.088040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:19.088003 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f21dc0b-5ee6-4300-88c4-906e6ba44047-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:19.088040 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:19.088040 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pg4\" (UniqueName: \"kubernetes.io/projected/3f21dc0b-5ee6-4300-88c4-906e6ba44047-kube-api-access-l6pg4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:19.683953 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:19.683921 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" Nov 01 05:59:19.684137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:19.683961 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj" event={"ID":"3f21dc0b-5ee6-4300-88c4-906e6ba44047","Type":"ContainerDied","Data":"5f2a661a32bf6feb5f062b7e95a02b7043cb6f4be3e49950b0b7079e024adb4a"} Nov 01 05:59:19.684137 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:19.684014 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2a661a32bf6feb5f062b7e95a02b7043cb6f4be3e49950b0b7079e024adb4a" Nov 01 05:59:20.383369 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:20.383325 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" containerID="cri-o://187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016" gracePeriod=25 Nov 01 05:59:20.489568 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:20.489529 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" containerID="cri-o://f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9" gracePeriod=25 Nov 01 05:59:21.403910 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.403869 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:21.404641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404621 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3f21dc0b-5ee6-4300-88c4-906e6ba44047" containerName="revoke-keys" Nov 01 05:59:21.404641 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404645 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f21dc0b-5ee6-4300-88c4-906e6ba44047" containerName="revoke-keys" Nov 01 05:59:21.404801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404703 4425 cpu_manager.go:414] "containerMap: removing container" podUID="36397df4-3586-4802-a7aa-88d303e3927e" containerName="revoke-keys" Nov 01 05:59:21.404801 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404711 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36397df4-3586-4802-a7aa-88d303e3927e" containerName="revoke-keys" Nov 01 05:59:21.404871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404852 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="36397df4-3586-4802-a7aa-88d303e3927e" containerName="revoke-keys" Nov 01 05:59:21.404871 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.404868 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f21dc0b-5ee6-4300-88c4-906e6ba44047" containerName="revoke-keys" Nov 01 05:59:21.408788 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.408771 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.416707 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.416685 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5746\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.416812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.416758 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.416812 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.416781 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.416969 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.416949 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.516570 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.516531 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:21.518031 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.518098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518056 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.518285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518245 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.518430 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518412 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5746\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.518920 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518891 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.519035 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.518889 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.520829 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.520809 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.612279 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.588632 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5746\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.722675 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:21.722586 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:21.733105 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733073 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:21.733105 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733108 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:21.733379 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733118 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:21.733379 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733126 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:21.733379 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733144 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:21.733379 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:21.733217 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:29.733197067 +0000 UTC m=+3384.873785963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found] Nov 01 05:59:22.123416 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:22.123388 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:22.123903 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:59:22.123879 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edf8432_a9c2_4b59_bc14_7aed4b507899.slice/crio-2ada69401805828e1b88c676e02d56c147124dd2198f698212eba4145005a790 WatchSource:0}: Error finding container 2ada69401805828e1b88c676e02d56c147124dd2198f698212eba4145005a790: Status 404 returned error can't find the container with id 2ada69401805828e1b88c676e02d56c147124dd2198f698212eba4145005a790 Nov 01 05:59:22.713072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:22.712872 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" event={"ID":"5edf8432-a9c2-4b59-bc14-7aed4b507899","Type":"ContainerStarted","Data":"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd"} Nov 01 05:59:22.713072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:22.712979 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" event={"ID":"5edf8432-a9c2-4b59-bc14-7aed4b507899","Type":"ContainerStarted","Data":"2ada69401805828e1b88c676e02d56c147124dd2198f698212eba4145005a790"} Nov 01 05:59:22.713072 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:22.713027 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:22.838110 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:22.838039 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" podStartSLOduration=1.8380196610000001 podStartE2EDuration="1.838019661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:22.830876494 +0000 UTC m=+3377.971465446" watchObservedRunningTime="2026-11-01 05:59:22.838019661 +0000 UTC m=+3377.978608584" Nov 01 05:59:26.000294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:26.000234 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:26.051325 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:26.051292 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:59:26.952315 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:26.952278 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 05:59:29.216989 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:29.216945 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:59:29.218631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:29.218594 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:59:29.781180 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781146 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-0a08dbd11a06: secret "agc-cred-0a08dbd11a06" not found Nov 01 05:59:29.781180 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781185 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-21c241f95e7c: secret "agc-cred-21c241f95e7c" not found Nov 01 05:59:29.781455 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781192 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-30604247b165: secret "agc-cred-30604247b165" not found Nov 01 05:59:29.781455 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781200 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-8fb022cc51dc: secret "agc-cred-8fb022cc51dc" not found Nov 01 05:59:29.781455 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781206 4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/e2e-cfl-eafa67-provider-cred: secret "e2e-cfl-eafa67-provider-cred" not found Nov 01 05:59:29.781455 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781217 4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw: [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found, secret "e2e-cfl-eafa67-provider-cred" not found] Nov 01 05:59:29.781455 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:29.781319 4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials podName:e50689af-8ebb-4e48-8f1c-0a28f2e5cb56 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:45.781300757 +0000 UTC m=+3400.921889656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56") : [secret "agc-cred-0a08dbd11a06" not found, secret "agc-cred-21c241f95e7c" not found, secret "agc-cred-30604247b165" not found, secret "agc-cred-8fb022cc51dc" not found, secret "e2e-cfl-eafa67-provider-cred" not found] Nov 01 05:59:31.001131 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.001058 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" podUID="5edf8432-a9c2-4b59-bc14-7aed4b507899" containerName="payload-processing" containerID="cri-o://66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd" gracePeriod=25 Nov 01 05:59:31.052353 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.052314 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" podUID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" containerName="payload-processing" containerID="cri-o://b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645" gracePeriod=25 Nov 01 05:59:31.344473 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.344445 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:31.377021 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.376992 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:59:31.503346 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503316 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\") pod \"5edf8432-a9c2-4b59-bc14-7aed4b507899\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " Nov 01 05:59:31.503519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503396 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\") pod \"5edf8432-a9c2-4b59-bc14-7aed4b507899\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " Nov 01 05:59:31.503519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503444 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\") pod \"5edf8432-a9c2-4b59-bc14-7aed4b507899\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " Nov 01 05:59:31.503519 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503482 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\") pod \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " Nov 01 05:59:31.503699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503532 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\") pod \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " Nov 01 05:59:31.503699 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503598 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\") pod \"5edf8432-a9c2-4b59-bc14-7aed4b507899\" (UID: \"5edf8432-a9c2-4b59-bc14-7aed4b507899\") " Nov 01 05:59:31.503834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503723 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume" pod "5edf8432-a9c2-4b59-bc14-7aed4b507899" (UID: "5edf8432-a9c2-4b59-bc14-7aed4b507899"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:31.503834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503740 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\") pod \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " Nov 01 05:59:31.503834 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.503790 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\") pod \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\" (UID: \"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56\") " Nov 01 05:59:31.504282 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.504202 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume" pod "5edf8432-a9c2-4b59-bc14-7aed4b507899" (UID: "5edf8432-a9c2-4b59-bc14-7aed4b507899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:31.504459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.504342 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.504459 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.504357 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5edf8432-a9c2-4b59-bc14-7aed4b507899-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.504578 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.504519 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume" pod "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:31.505007 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.504979 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume" pod "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:31.506024 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.506000 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials" pod "5edf8432-a9c2-4b59-bc14-7aed4b507899" (UID: "5edf8432-a9c2-4b59-bc14-7aed4b507899"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:31.506146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.506024 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746" pod "5edf8432-a9c2-4b59-bc14-7aed4b507899" (UID: "5edf8432-a9c2-4b59-bc14-7aed4b507899"). InnerVolumeSpecName "kube-api-access-w5746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:31.506146 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.506118 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm" pod "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56"). InnerVolumeSpecName "kube-api-access-4zgkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:31.506631 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.506608 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials" pod "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" (UID: "e50689af-8ebb-4e48-8f1c-0a28f2e5cb56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:31.606201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606164 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5746\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-kube-api-access-w5746\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.606201 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606203 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5edf8432-a9c2-4b59-bc14-7aed4b507899-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.606499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606217 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zgkm\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-kube-api-access-4zgkm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.606499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606232 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.606499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606246 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.606499 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.606281 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:31.816268 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.816209 4425 generic.go:356] "Generic (PLEG): container finished" podID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" containerID="b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645" exitCode=0 Nov 01 05:59:31.816453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.816281 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" Nov 01 05:59:31.816453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.816291 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" event={"ID":"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56","Type":"ContainerDied","Data":"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645"} Nov 01 05:59:31.816453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.816358 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw" event={"ID":"e50689af-8ebb-4e48-8f1c-0a28f2e5cb56","Type":"ContainerDied","Data":"5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92"} Nov 01 05:59:31.816453 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.816382 4425 scope.go:122] "RemoveContainer" containerID="b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645" Nov 01 05:59:31.818240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.818208 4425 generic.go:356] "Generic (PLEG): container finished" podID="5edf8432-a9c2-4b59-bc14-7aed4b507899" containerID="66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd" exitCode=0 Nov 01 05:59:31.818240 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.818238 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" event={"ID":"5edf8432-a9c2-4b59-bc14-7aed4b507899","Type":"ContainerDied","Data":"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd"} Nov 01 05:59:31.818486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.818299 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" event={"ID":"5edf8432-a9c2-4b59-bc14-7aed4b507899","Type":"ContainerDied","Data":"2ada69401805828e1b88c676e02d56c147124dd2198f698212eba4145005a790"} Nov 01 05:59:31.818486 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.818373 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x" Nov 01 05:59:31.831365 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.831341 4425 scope.go:122] "RemoveContainer" containerID="b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645" Nov 01 05:59:31.831672 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:31.831647 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645\": container with ID starting with b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645 not found: ID does not exist" containerID="b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645" Nov 01 05:59:31.831746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.831688 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645"} err="failed to get container status \"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645\": rpc error: code = NotFound desc = could not find container \"b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645\": container with ID starting with b8bdeec4e66d481c5ebea140cdb51cb6c0296350a0267a27d9feceaeda744645 not found: ID does not exist" Nov 01 05:59:31.831746 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.831714 4425 scope.go:122] "RemoveContainer" containerID="66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd" Nov 01 05:59:31.858577 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:31.858505 4425 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50689af_8ebb_4e48_8f1c_0a28f2e5cb56.slice/crio-5762533961a8cf529095cc86acfaf8a5408ca6067a29d62acd0da64efc320d92\": RecentStats: unable to find data in memory cache]" Nov 01 05:59:31.863725 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.863700 4425 scope.go:122] "RemoveContainer" containerID="66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd" Nov 01 05:59:31.864124 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:31.864096 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd\": container with ID starting with 66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd not found: ID does not exist" containerID="66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd" Nov 01 05:59:31.864221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.864130 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd"} err="failed to get container status \"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd\": rpc error: code = NotFound desc = could not find container \"66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd\": container with ID starting with 66a46f2e8e57658b4a521b0d7aa06aa5186c3f843d769770a67023ebc25ffccd not found: ID does not exist" Nov 01 05:59:31.984860 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.984825 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:59:31.988163 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:31.988140 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-68574jlpw"] Nov 01 05:59:32.098558 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:32.098527 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:32.104372 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:32.104346 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-f854rhg9x"] Nov 01 05:59:32.979061 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:32.979005 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edf8432-a9c2-4b59-bc14-7aed4b507899" path="/var/lib/kubelet/pods/5edf8432-a9c2-4b59-bc14-7aed4b507899/volumes" Nov 01 05:59:32.980963 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:32.980935 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" path="/var/lib/kubelet/pods/e50689af-8ebb-4e48-8f1c-0a28f2e5cb56/volumes" Nov 01 05:59:39.216051 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:39.216014 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:59:39.216491 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:39.216011 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 05:59:40.680722 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.680687 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:59:40.752098 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.752065 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\" (UniqueName: \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\") pod \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " Nov 01 05:59:40.752584 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.752548 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\") pod \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\" (UID: \"cea92693-1c83-4e04-a4aa-7d0a1ab69159\") " Nov 01 05:59:40.753199 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.753067 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume" pod "cea92693-1c83-4e04-a4aa-7d0a1ab69159" (UID: "cea92693-1c83-4e04-a4aa-7d0a1ab69159"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:40.753886 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.753864 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea92693-1c83-4e04-a4aa-7d0a1ab69159-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:40.758885 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.758858 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd" pod "cea92693-1c83-4e04-a4aa-7d0a1ab69159" (UID: "cea92693-1c83-4e04-a4aa-7d0a1ab69159"). InnerVolumeSpecName "kube-api-access-5ndpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:40.811285 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.811216 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:59:40.855431 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.855399 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\" (UniqueName: \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\") pod \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " Nov 01 05:59:40.855637 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.855447 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\") pod \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\" (UID: \"8ce603a8-4361-4c41-b09a-04d84a69b5d5\") " Nov 01 05:59:40.855862 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.855838 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume" pod "8ce603a8-4361-4c41-b09a-04d84a69b5d5" (UID: "8ce603a8-4361-4c41-b09a-04d84a69b5d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 05:59:40.856080 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.856066 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndpd\" (UniqueName: \"kubernetes.io/projected/cea92693-1c83-4e04-a4aa-7d0a1ab69159-kube-api-access-5ndpd\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:40.856132 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.856084 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce603a8-4361-4c41-b09a-04d84a69b5d5-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:40.857755 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.857738 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb" pod "8ce603a8-4361-4c41-b09a-04d84a69b5d5" (UID: "8ce603a8-4361-4c41-b09a-04d84a69b5d5"). InnerVolumeSpecName "kube-api-access-sxspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 05:59:40.904160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.904085 4425 generic.go:356] "Generic (PLEG): container finished" podID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerID="f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9" exitCode=0 Nov 01 05:59:40.904160 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.904138 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" Nov 01 05:59:40.904414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.904176 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" event={"ID":"8ce603a8-4361-4c41-b09a-04d84a69b5d5","Type":"ContainerDied","Data":"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9"} Nov 01 05:59:40.904414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.904235 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d" event={"ID":"8ce603a8-4361-4c41-b09a-04d84a69b5d5","Type":"ContainerDied","Data":"0f5c89d7f25bffcf057581add0ba2f7e670dfd16bfdbab508ad6c44822385266"} Nov 01 05:59:40.904414 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.904289 4425 scope.go:122] "RemoveContainer" containerID="f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9" Nov 01 05:59:40.905704 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.905678 4425 generic.go:356] "Generic (PLEG): container finished" podID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerID="187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016" exitCode=0 Nov 01 05:59:40.905853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.905726 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" Nov 01 05:59:40.905853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.905758 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" event={"ID":"cea92693-1c83-4e04-a4aa-7d0a1ab69159","Type":"ContainerDied","Data":"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016"} Nov 01 05:59:40.905853 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.905810 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw" event={"ID":"cea92693-1c83-4e04-a4aa-7d0a1ab69159","Type":"ContainerDied","Data":"38dfdd831462c880ed7d311f073d42d41780f7771fb55eb4438d4c98ec980238"} Nov 01 05:59:40.916352 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.916311 4425 scope.go:122] "RemoveContainer" containerID="f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9" Nov 01 05:59:40.916903 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:40.916744 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9\": container with ID starting with f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9 not found: ID does not exist" containerID="f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9" Nov 01 05:59:40.916903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.916780 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9"} err="failed to get container status \"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9\": rpc error: code = NotFound desc = could not find container \"f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9\": container with ID starting with f4371c00ae9b0f43118874b861a07b78498069a384feb317819740d819e7f0c9 not found: ID does not exist" Nov 01 05:59:40.916903 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.916804 4425 scope.go:122] "RemoveContainer" containerID="187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016" Nov 01 05:59:40.941494 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.941369 4425 scope.go:122] "RemoveContainer" containerID="187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016" Nov 01 05:59:40.965294 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:40.941760 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016\": container with ID starting with 187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016 not found: ID does not exist" containerID="187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016" Nov 01 05:59:40.965294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.941796 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016"} err="failed to get container status \"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016\": rpc error: code = NotFound desc = could not find container \"187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016\": container with ID starting with 187c614084c8642a5b007ebfa1a58f71fff8acd7bd4263808b153f41e1a1e016 not found: ID does not exist" Nov 01 05:59:40.965294 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:40.957807 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxspb\" (UniqueName: \"kubernetes.io/projected/8ce603a8-4361-4c41-b09a-04d84a69b5d5-kube-api-access-sxspb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 05:59:41.073387 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:41.073350 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:59:41.076693 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:41.076670 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d70ff-5df597f9cf-7v76d"] Nov 01 05:59:41.188182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:41.188095 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:59:41.191781 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:41.191756 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d70ff-6dc999899-r9nvw"] Nov 01 05:59:42.974159 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:42.974125 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" path="/var/lib/kubelet/pods/8ce603a8-4361-4c41-b09a-04d84a69b5d5/volumes" Nov 01 05:59:42.974604 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:42.974589 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" path="/var/lib/kubelet/pods/cea92693-1c83-4e04-a4aa-7d0a1ab69159/volumes" Nov 01 05:59:48.452111 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452072 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 05:59:48.452905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452881 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" containerName="payload-processing" Nov 01 05:59:48.452905 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452904 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452917 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452922 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452933 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5edf8432-a9c2-4b59-bc14-7aed4b507899" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452938 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edf8432-a9c2-4b59-bc14-7aed4b507899" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452962 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.452967 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.453054 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea92693-1c83-4e04-a4aa-7d0a1ab69159" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.453067 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50689af-8ebb-4e48-8f1c-0a28f2e5cb56" containerName="payload-processing" Nov 01 05:59:48.453070 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.453082 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce603a8-4361-4c41-b09a-04d84a69b5d5" containerName="payload-pre-processing" Nov 01 05:59:48.453471 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.453090 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edf8432-a9c2-4b59-bc14-7aed4b507899" containerName="payload-processing" Nov 01 05:59:48.458323 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.458302 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.513765 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.513733 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 05:59:48.559766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.559706 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"openshift-service-ca.crt\"" Nov 01 05:59:48.559766 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.559753 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"agc-cred-f7650e0802c8\"" Nov 01 05:59:48.560028 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.559866 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"kube-root-ca.crt\"" Nov 01 05:59:48.560221 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.560197 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.560356 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.560294 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.560434 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.560412 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8d8\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.560518 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.560500 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.563275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.563227 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"routing-overlay\"" Nov 01 05:59:48.563275 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.563232 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-098e4d\"" Nov 01 05:59:48.563427 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.563298 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-098e4d\"/\"payload-processing-external-model-e2e--989e5a12-dockercfg-4lpn5\"" Nov 01 05:59:48.661943 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.661909 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.662182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.661971 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.662182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.662024 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8d8\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.662182 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.662074 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.662767 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.662739 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.662861 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.662784 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.664745 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.664723 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.724239 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.724166 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8d8\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\") pod \"payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:48.773130 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:48.772780 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:49.199955 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:49.199925 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 05:59:49.200292 ip-10-0-141-97 kubenswrapper[4425]: W1101 05:59:49.200247 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bc8339_7c9f_4b0f_8dc9_059dcabfc531.slice/crio-ca901eec759b79e6a3b71f5b9479c68873aec0357905c95cb2f84954be137262 WatchSource:0}: Error finding container ca901eec759b79e6a3b71f5b9479c68873aec0357905c95cb2f84954be137262: Status 404 returned error can't find the container with id ca901eec759b79e6a3b71f5b9479c68873aec0357905c95cb2f84954be137262 Nov 01 05:59:49.974585 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:49.974546 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" event={"ID":"54bc8339-7c9f-4b0f-8dc9-059dcabfc531","Type":"ContainerStarted","Data":"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c"} Nov 01 05:59:49.975001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:49.974599 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" event={"ID":"54bc8339-7c9f-4b0f-8dc9-059dcabfc531","Type":"ContainerStarted","Data":"ca901eec759b79e6a3b71f5b9479c68873aec0357905c95cb2f84954be137262"} Nov 01 05:59:49.975001 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:49.974646 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 05:59:50.095512 ip-10-0-141-97 kubenswrapper[4425]: I1101 05:59:50.095456 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" podStartSLOduration=2.095442875 podStartE2EDuration="2.095442875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:50.092190906 +0000 UTC m=+3405.232779825" watchObservedRunningTime="2026-11-01 05:59:50.095442875 +0000 UTC m=+3405.236031793" Nov 01 05:59:56.967519 ip-10-0-141-97 kubenswrapper[4425]: E1101 05:59:56.967481 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:00:00.170718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.170686 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:00.186203 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.186164 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:00.186406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.186324 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.190404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.190383 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:00.190563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.190500 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.194231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.194210 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.221338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.221305 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:00.226493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.226473 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:00.226605 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.226586 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.231438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.231418 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:00.231543 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.231527 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.236454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.236431 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:00.236581 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.236540 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.242653 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.242633 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:00.242762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.242741 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.247478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.247444 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:00.247585 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.247556 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.252191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.252171 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.268335 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.268305 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:00:00.268335 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.268325 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:00:00.268493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.268395 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:00:00.274849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.274825 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:00.309672 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309641 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.309933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309705 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvh7\" (UniqueName: \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.309933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309843 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rls49\" (UniqueName: \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.309933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309907 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkm8\" (UniqueName: \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.309933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309934 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl42q\" (UniqueName: \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309952 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ppj\" (UniqueName: \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.309980 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310056 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkp8h\" (UniqueName: \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310089 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4n7q\" (UniqueName: \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310107 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310139 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxnp7\" (UniqueName: \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310183 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.310241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310221 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.310660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310322 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.310660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310411 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.310660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310450 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p8k\" (UniqueName: \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.310660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310484 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.310660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.310501 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.326280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.322218 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:00.328661 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.328630 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:00.333715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.333683 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:00.377840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.377218 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:00.381957 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.381932 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:00.389347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.389320 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:00.401462 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.401424 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:00.411214 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411182 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411221 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxnp7\" (UniqueName: \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411269 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411295 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411332 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411384 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.411441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411419 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p8k\" (UniqueName: \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.411825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411461 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.411825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411488 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.411825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411522 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.411825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.411572 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvh7\" (UniqueName: \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.412145 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412115 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.412237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412165 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.412237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412200 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.412384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412288 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rls49\" (UniqueName: \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.412384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412349 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.412518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412354 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkm8\" (UniqueName: \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.412581 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412542 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl42q\" (UniqueName: \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.412581 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412556 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.412720 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ppj\" (UniqueName: \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.412720 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412619 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.412841 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412732 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkp8h\" (UniqueName: \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.412841 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412781 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n7q\" (UniqueName: \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.413024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412901 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.413024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.412902 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.413623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.413592 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.413723 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.413653 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.452609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.452532 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:00.474198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.473847 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p8k\" (UniqueName: \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.474198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.474136 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkm8\" (UniqueName: \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\") pod \"maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.475684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.475380 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxnp7\" (UniqueName: \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\") pod \"maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.475684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.475493 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvh7\" (UniqueName: \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.476183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.476161 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkp8h\" (UniqueName: \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\") pod \"maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.476439 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.476415 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl42q\" (UniqueName: \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\") pod \"maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.477189 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.477163 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ppj\" (UniqueName: \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\") pod \"maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.478086 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.478066 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rls49\" (UniqueName: \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.478674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.478648 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n7q\" (UniqueName: \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\") pod \"maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.502925 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.502900 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:00.511346 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.511323 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:00.518337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.518104 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:00.548439 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.548117 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:00.552375 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.552201 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:00.560650 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.560622 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:00.571971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.571878 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:00.579714 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.579677 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:00.597231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.596531 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:00.908044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.907836 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:00.909972 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:00.909939 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da493b6_987b_4011_8ffe_42f1207d4a4a.slice/crio-8be5f29e424450aa0e53fa3bbc3384952b2e598fa6a8b7498880ec7cb88e8bed WatchSource:0}: Error finding container 8be5f29e424450aa0e53fa3bbc3384952b2e598fa6a8b7498880ec7cb88e8bed: Status 404 returned error can't find the container with id 8be5f29e424450aa0e53fa3bbc3384952b2e598fa6a8b7498880ec7cb88e8bed Nov 01 06:00:00.989738 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:00.989665 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 06:00:01.053642 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:01.053608 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967255e3_b934_487e_a9a0_ff3b1756a98b.slice/crio-c76a708517700bade037cd1f980e294360015603348219ad046738923ff8e62d WatchSource:0}: Error finding container c76a708517700bade037cd1f980e294360015603348219ad046738923ff8e62d: Status 404 returned error can't find the container with id c76a708517700bade037cd1f980e294360015603348219ad046738923ff8e62d Nov 01 06:00:01.054227 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.054201 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:01.059319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.059293 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" event={"ID":"8da493b6-987b-4011-8ffe-42f1207d4a4a","Type":"ContainerStarted","Data":"8be5f29e424450aa0e53fa3bbc3384952b2e598fa6a8b7498880ec7cb88e8bed"} Nov 01 06:00:01.104483 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.104449 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:01.113748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.113497 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:01.121180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.121155 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:01.127908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.127883 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:01.154880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.154838 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:01.183209 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.183178 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:01.184134 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:01.184102 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525dc9a9_3201_4e77_be83_2d33062a6a9d.slice/crio-606cc8aa45de523101c366863eb706dbe3bdeb251995c23802e2870d04a83913 WatchSource:0}: Error finding container 606cc8aa45de523101c366863eb706dbe3bdeb251995c23802e2870d04a83913: Status 404 returned error can't find the container with id 606cc8aa45de523101c366863eb706dbe3bdeb251995c23802e2870d04a83913 Nov 01 06:00:01.272155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:01.272130 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:01.446411 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:01.446369 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09037267_e69f_43e8_8ba5_d53b3dc179a0.slice/crio-10582decc235395ff0b6a51f2a4d315548ea787b594cc5a03b343ab84150b164 WatchSource:0}: Error finding container 10582decc235395ff0b6a51f2a4d315548ea787b594cc5a03b343ab84150b164: Status 404 returned error can't find the container with id 10582decc235395ff0b6a51f2a4d315548ea787b594cc5a03b343ab84150b164 Nov 01 06:00:02.069053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.069016 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerStarted","Data":"6df0ad8576571ca028585247d58183d4845fd708393d4229ba2338c1242a5c69"} Nov 01 06:00:02.069225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.069084 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerStarted","Data":"ab5db61975847311d3a5848edf8d16914d94b0b119891e279f47f2652c7ea14d"} Nov 01 06:00:02.070928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.070898 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerStarted","Data":"ce65675c4ce2f7296076fe7d14788347829c9ff10e61ca9124c61ff51774dbc6"} Nov 01 06:00:02.071022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.070951 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerStarted","Data":"c76a708517700bade037cd1f980e294360015603348219ad046738923ff8e62d"} Nov 01 06:00:02.072733 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.072707 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerStarted","Data":"8c2adbf9f9af4d7af369af69a058b917f94ffde97e49fb4941982e567d99bd15"} Nov 01 06:00:02.072904 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.072885 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerStarted","Data":"606cc8aa45de523101c366863eb706dbe3bdeb251995c23802e2870d04a83913"} Nov 01 06:00:02.074229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.074203 4425 generic.go:356] "Generic (PLEG): container finished" podID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerID="d5cf0a8c6ef0c6873759b914228500f14349295df45b7b8d711086bcfb1723cf" exitCode=60 Nov 01 06:00:02.074358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.074295 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" event={"ID":"8da493b6-987b-4011-8ffe-42f1207d4a4a","Type":"ContainerDied","Data":"d5cf0a8c6ef0c6873759b914228500f14349295df45b7b8d711086bcfb1723cf"} Nov 01 06:00:02.074486 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.074470 4425 scope.go:122] "RemoveContainer" containerID="d5cf0a8c6ef0c6873759b914228500f14349295df45b7b8d711086bcfb1723cf" Nov 01 06:00:02.075985 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.075968 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerStarted","Data":"5319906d224440f695c204a593784d9fdf08f8ad07e0d2bbd3f7b232fdfae599"} Nov 01 06:00:02.076053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.076001 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerStarted","Data":"40932c95a6a04671993a1ce763bdab571e360fa7560d79aff4596231f198f7d0"} Nov 01 06:00:02.077749 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.077676 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerStarted","Data":"8c4bf262a9fe323ba9a6512446def72483d7250d9dea05ee33fe15565b81a14a"} Nov 01 06:00:02.077749 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.077711 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerStarted","Data":"bcba5791ba627cf4b998b37ea12dee456c7d3771560a3c8373fac2142cec7c76"} Nov 01 06:00:02.079130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.079101 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerStarted","Data":"10582decc235395ff0b6a51f2a4d315548ea787b594cc5a03b343ab84150b164"} Nov 01 06:00:02.080719 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.080690 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerStarted","Data":"34391156bc2391a66cc6bd563a4faa7416bf68626d6885547a80b095b9bae101"} Nov 01 06:00:02.080826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.080734 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerStarted","Data":"edfee5907b91b79bef09da4067cac1d7331e7528d51db3718cff0a8c03f7e975"} Nov 01 06:00:02.082229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.082211 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerStarted","Data":"3f7a9b121d198ffcb2afa41414e19b40b9329bd4f8e0f8de9a17e96ce8e65e55"} Nov 01 06:00:02.082334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.082243 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerStarted","Data":"82ebb19b78cb6226e817e73b13f5740483ba558c552343bf3de516323be29daf"} Nov 01 06:00:02.187045 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.186990 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" podStartSLOduration=2.186971334 podStartE2EDuration="2.186971334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.184722047 +0000 UTC m=+3417.325310988" watchObservedRunningTime="2026-11-01 06:00:02.186971334 +0000 UTC m=+3417.327560255" Nov 01 06:00:02.302721 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.302664 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" podStartSLOduration=2.302651192 podStartE2EDuration="2.302651192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.301332895 +0000 UTC m=+3417.441921814" watchObservedRunningTime="2026-11-01 06:00:02.302651192 +0000 UTC m=+3417.443240111" Nov 01 06:00:02.420711 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.420658 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" podStartSLOduration=2.420643407 podStartE2EDuration="2.420643407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.417308233 +0000 UTC m=+3417.557897185" watchObservedRunningTime="2026-11-01 06:00:02.420643407 +0000 UTC m=+3417.561232327" Nov 01 06:00:02.533692 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.533639 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" podStartSLOduration=2.533622818 podStartE2EDuration="2.533622818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.532985569 +0000 UTC m=+3417.673574526" watchObservedRunningTime="2026-11-01 06:00:02.533622818 +0000 UTC m=+3417.674211736" Nov 01 06:00:02.651343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.651291 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" podStartSLOduration=2.651276428 podStartE2EDuration="2.651276428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.648629875 +0000 UTC m=+3417.789218793" watchObservedRunningTime="2026-11-01 06:00:02.651276428 +0000 UTC m=+3417.791865359" Nov 01 06:00:02.881875 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.881783 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" podStartSLOduration=2.881768573 podStartE2EDuration="2.881768573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.879987344 +0000 UTC m=+3418.020576264" watchObservedRunningTime="2026-11-01 06:00:02.881768573 +0000 UTC m=+3418.022357491" Nov 01 06:00:02.999722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:02.999658 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" podStartSLOduration=2.999639646 podStartE2EDuration="2.999639646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.995464939 +0000 UTC m=+3418.136053868" watchObservedRunningTime="2026-11-01 06:00:02.999639646 +0000 UTC m=+3418.140228566" Nov 01 06:00:03.091777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.091736 4425 generic.go:356] "Generic (PLEG): container finished" podID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerID="34391156bc2391a66cc6bd563a4faa7416bf68626d6885547a80b095b9bae101" exitCode=60 Nov 01 06:00:03.091969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.091825 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerDied","Data":"34391156bc2391a66cc6bd563a4faa7416bf68626d6885547a80b095b9bae101"} Nov 01 06:00:03.092177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.092160 4425 scope.go:122] "RemoveContainer" containerID="34391156bc2391a66cc6bd563a4faa7416bf68626d6885547a80b095b9bae101" Nov 01 06:00:03.093658 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.093634 4425 generic.go:356] "Generic (PLEG): container finished" podID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerID="3f7a9b121d198ffcb2afa41414e19b40b9329bd4f8e0f8de9a17e96ce8e65e55" exitCode=60 Nov 01 06:00:03.093761 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.093711 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerDied","Data":"3f7a9b121d198ffcb2afa41414e19b40b9329bd4f8e0f8de9a17e96ce8e65e55"} Nov 01 06:00:03.093983 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.093964 4425 scope.go:122] "RemoveContainer" containerID="3f7a9b121d198ffcb2afa41414e19b40b9329bd4f8e0f8de9a17e96ce8e65e55" Nov 01 06:00:03.095752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.095614 4425 generic.go:356] "Generic (PLEG): container finished" podID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerID="6df0ad8576571ca028585247d58183d4845fd708393d4229ba2338c1242a5c69" exitCode=60 Nov 01 06:00:03.095752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.095693 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerDied","Data":"6df0ad8576571ca028585247d58183d4845fd708393d4229ba2338c1242a5c69"} Nov 01 06:00:03.095922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.095906 4425 scope.go:122] "RemoveContainer" containerID="6df0ad8576571ca028585247d58183d4845fd708393d4229ba2338c1242a5c69" Nov 01 06:00:03.097505 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.097488 4425 generic.go:356] "Generic (PLEG): container finished" podID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerID="ce65675c4ce2f7296076fe7d14788347829c9ff10e61ca9124c61ff51774dbc6" exitCode=60 Nov 01 06:00:03.097558 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.097545 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerDied","Data":"ce65675c4ce2f7296076fe7d14788347829c9ff10e61ca9124c61ff51774dbc6"} Nov 01 06:00:03.097780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.097767 4425 scope.go:122] "RemoveContainer" containerID="ce65675c4ce2f7296076fe7d14788347829c9ff10e61ca9124c61ff51774dbc6" Nov 01 06:00:03.099305 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.099281 4425 generic.go:356] "Generic (PLEG): container finished" podID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerID="8c2adbf9f9af4d7af369af69a058b917f94ffde97e49fb4941982e567d99bd15" exitCode=60 Nov 01 06:00:03.099382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.099354 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerDied","Data":"8c2adbf9f9af4d7af369af69a058b917f94ffde97e49fb4941982e567d99bd15"} Nov 01 06:00:03.099586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.099573 4425 scope.go:122] "RemoveContainer" containerID="8c2adbf9f9af4d7af369af69a058b917f94ffde97e49fb4941982e567d99bd15" Nov 01 06:00:03.101776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.101754 4425 generic.go:356] "Generic (PLEG): container finished" podID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerID="97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556" exitCode=60 Nov 01 06:00:03.101858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.101829 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" event={"ID":"8da493b6-987b-4011-8ffe-42f1207d4a4a","Type":"ContainerDied","Data":"97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556"} Nov 01 06:00:03.101912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.101870 4425 scope.go:122] "RemoveContainer" containerID="d5cf0a8c6ef0c6873759b914228500f14349295df45b7b8d711086bcfb1723cf" Nov 01 06:00:03.102037 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.102021 4425 scope.go:122] "RemoveContainer" containerID="97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556" Nov 01 06:00:03.102314 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:03.102291 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p_odh-ai-gateway-infra(8da493b6-987b-4011-8ffe-42f1207d4a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" Nov 01 06:00:03.104118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.104095 4425 generic.go:356] "Generic (PLEG): container finished" podID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerID="5319906d224440f695c204a593784d9fdf08f8ad07e0d2bbd3f7b232fdfae599" exitCode=60 Nov 01 06:00:03.104358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.104330 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerDied","Data":"5319906d224440f695c204a593784d9fdf08f8ad07e0d2bbd3f7b232fdfae599"} Nov 01 06:00:03.104519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.104503 4425 scope.go:122] "RemoveContainer" containerID="5319906d224440f695c204a593784d9fdf08f8ad07e0d2bbd3f7b232fdfae599" Nov 01 06:00:03.106805 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.106788 4425 generic.go:356] "Generic (PLEG): container finished" podID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerID="8c4bf262a9fe323ba9a6512446def72483d7250d9dea05ee33fe15565b81a14a" exitCode=60 Nov 01 06:00:03.106919 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.106894 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerDied","Data":"8c4bf262a9fe323ba9a6512446def72483d7250d9dea05ee33fe15565b81a14a"} Nov 01 06:00:03.107127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.107050 4425 scope.go:122] "RemoveContainer" containerID="8c4bf262a9fe323ba9a6512446def72483d7250d9dea05ee33fe15565b81a14a" Nov 01 06:00:03.109566 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.109545 4425 generic.go:356] "Generic (PLEG): container finished" podID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerID="7645ddaef245b9b83975d679bbcd7d44bdfdde7b00e6f577b230d9623cc77b17" exitCode=60 Nov 01 06:00:03.109666 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.109577 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerDied","Data":"7645ddaef245b9b83975d679bbcd7d44bdfdde7b00e6f577b230d9623cc77b17"} Nov 01 06:00:03.110191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:03.109829 4425 scope.go:122] "RemoveContainer" containerID="7645ddaef245b9b83975d679bbcd7d44bdfdde7b00e6f577b230d9623cc77b17" Nov 01 06:00:04.118364 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.118326 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerStarted","Data":"1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54"} Nov 01 06:00:04.120541 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.120507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerStarted","Data":"fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22"} Nov 01 06:00:04.122447 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.122422 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerStarted","Data":"047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab"} Nov 01 06:00:04.124312 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.124294 4425 scope.go:122] "RemoveContainer" containerID="97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556" Nov 01 06:00:04.124565 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:04.124545 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p_odh-ai-gateway-infra(8da493b6-987b-4011-8ffe-42f1207d4a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" Nov 01 06:00:04.125950 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.125929 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerStarted","Data":"c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715"} Nov 01 06:00:04.127848 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.127827 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerStarted","Data":"ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713"} Nov 01 06:00:04.129735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.129715 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerStarted","Data":"35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300"} Nov 01 06:00:04.131643 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.131623 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerStarted","Data":"5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658"} Nov 01 06:00:04.133514 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:04.133493 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerStarted","Data":"ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088"} Nov 01 06:00:05.076738 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.076688 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" podStartSLOduration=5.076673438 podStartE2EDuration="5.076673438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:05.075757215 +0000 UTC m=+3420.216346133" watchObservedRunningTime="2026-11-01 06:00:05.076673438 +0000 UTC m=+3420.217262356" Nov 01 06:00:05.143140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.143106 4425 generic.go:356] "Generic (PLEG): container finished" podID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerID="c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715" exitCode=60 Nov 01 06:00:05.143619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.143165 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerDied","Data":"c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715"} Nov 01 06:00:05.143619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.143236 4425 scope.go:122] "RemoveContainer" containerID="5319906d224440f695c204a593784d9fdf08f8ad07e0d2bbd3f7b232fdfae599" Nov 01 06:00:05.143619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.143501 4425 scope.go:122] "RemoveContainer" containerID="c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715" Nov 01 06:00:05.143857 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.143834 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw_odh-ai-gateway-infra(e0b14236-a698-44d9-b884-4b74b20a7ddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" Nov 01 06:00:05.145517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.145494 4425 generic.go:356] "Generic (PLEG): container finished" podID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerID="ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713" exitCode=60 Nov 01 06:00:05.145629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.145539 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerDied","Data":"ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713"} Nov 01 06:00:05.145858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.145843 4425 scope.go:122] "RemoveContainer" containerID="ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713" Nov 01 06:00:05.146168 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.146127 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd_odh-ai-gateway-infra(a69944cd-44b1-4307-9680-76a3f62b86c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" Nov 01 06:00:05.147693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.147671 4425 generic.go:356] "Generic (PLEG): container finished" podID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerID="35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300" exitCode=60 Nov 01 06:00:05.147817 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.147759 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerDied","Data":"35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300"} Nov 01 06:00:05.147985 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.147957 4425 scope.go:122] "RemoveContainer" containerID="35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300" Nov 01 06:00:05.148238 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.148217 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt_odh-ai-gateway-infra(09037267-e69f-43e8-8ba5-d53b3dc179a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" Nov 01 06:00:05.149903 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.149884 4425 generic.go:356] "Generic (PLEG): container finished" podID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerID="5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658" exitCode=60 Nov 01 06:00:05.150021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.149962 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerDied","Data":"5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658"} Nov 01 06:00:05.150176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.150160 4425 scope.go:122] "RemoveContainer" containerID="5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658" Nov 01 06:00:05.150416 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.150395 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm_odh-ai-gateway-infra(12762aaf-20be-4b5a-8bf3-3aab4cdfcc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" Nov 01 06:00:05.152096 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.152064 4425 generic.go:356] "Generic (PLEG): container finished" podID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerID="ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088" exitCode=60 Nov 01 06:00:05.152176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.152113 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerDied","Data":"ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088"} Nov 01 06:00:05.152382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.152369 4425 scope.go:122] "RemoveContainer" containerID="ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088" Nov 01 06:00:05.152597 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.152569 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6_odh-ai-gateway-infra(6e832485-6f1e-4a56-b8ce-16e25987de8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" Nov 01 06:00:05.154134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.154117 4425 generic.go:356] "Generic (PLEG): container finished" podID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerID="1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54" exitCode=60 Nov 01 06:00:05.154269 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.154187 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerDied","Data":"1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54"} Nov 01 06:00:05.154416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.154398 4425 scope.go:122] "RemoveContainer" containerID="1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54" Nov 01 06:00:05.154624 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.154603 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4_odh-ai-gateway-infra(b0085083-fe55-40bf-a57f-cb544eb6be0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" Nov 01 06:00:05.156362 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.156342 4425 generic.go:356] "Generic (PLEG): container finished" podID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerID="fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22" exitCode=60 Nov 01 06:00:05.156480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.156398 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerDied","Data":"fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22"} Nov 01 06:00:05.156623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.156607 4425 scope.go:122] "RemoveContainer" containerID="fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22" Nov 01 06:00:05.156858 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.156835 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7_odh-ai-gateway-infra(967255e3-b934-487e-a9a0-ff3b1756a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" Nov 01 06:00:05.158417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.158239 4425 generic.go:356] "Generic (PLEG): container finished" podID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerID="047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab" exitCode=60 Nov 01 06:00:05.158417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.158284 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerDied","Data":"047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab"} Nov 01 06:00:05.158521 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.158435 4425 scope.go:122] "RemoveContainer" containerID="8c4bf262a9fe323ba9a6512446def72483d7250d9dea05ee33fe15565b81a14a" Nov 01 06:00:05.158521 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.158464 4425 scope.go:122] "RemoveContainer" containerID="047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab" Nov 01 06:00:05.158680 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:05.158662 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl_odh-ai-gateway-infra(525dc9a9-3201-4e77-be83-2d33062a6a9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" Nov 01 06:00:05.171960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.171939 4425 scope.go:122] "RemoveContainer" containerID="7645ddaef245b9b83975d679bbcd7d44bdfdde7b00e6f577b230d9623cc77b17" Nov 01 06:00:05.186508 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.186484 4425 scope.go:122] "RemoveContainer" containerID="34391156bc2391a66cc6bd563a4faa7416bf68626d6885547a80b095b9bae101" Nov 01 06:00:05.200822 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.200801 4425 scope.go:122] "RemoveContainer" containerID="3f7a9b121d198ffcb2afa41414e19b40b9329bd4f8e0f8de9a17e96ce8e65e55" Nov 01 06:00:05.213978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.213955 4425 scope.go:122] "RemoveContainer" containerID="6df0ad8576571ca028585247d58183d4845fd708393d4229ba2338c1242a5c69" Nov 01 06:00:05.227055 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.227036 4425 scope.go:122] "RemoveContainer" containerID="ce65675c4ce2f7296076fe7d14788347829c9ff10e61ca9124c61ff51774dbc6" Nov 01 06:00:05.240010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:05.239993 4425 scope.go:122] "RemoveContainer" containerID="8c2adbf9f9af4d7af369af69a058b917f94ffde97e49fb4941982e567d99bd15" Nov 01 06:00:06.169053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.169017 4425 scope.go:122] "RemoveContainer" containerID="fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22" Nov 01 06:00:06.169550 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.169439 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7_odh-ai-gateway-infra(967255e3-b934-487e-a9a0-ff3b1756a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" Nov 01 06:00:06.170989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.170970 4425 scope.go:122] "RemoveContainer" containerID="047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab" Nov 01 06:00:06.171218 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.171200 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl_odh-ai-gateway-infra(525dc9a9-3201-4e77-be83-2d33062a6a9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" Nov 01 06:00:06.172850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.172830 4425 scope.go:122] "RemoveContainer" containerID="c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715" Nov 01 06:00:06.173110 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.173079 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw_odh-ai-gateway-infra(e0b14236-a698-44d9-b884-4b74b20a7ddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" Nov 01 06:00:06.174633 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.174612 4425 scope.go:122] "RemoveContainer" containerID="ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713" Nov 01 06:00:06.174844 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.174825 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd_odh-ai-gateway-infra(a69944cd-44b1-4307-9680-76a3f62b86c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" Nov 01 06:00:06.176345 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.176326 4425 scope.go:122] "RemoveContainer" containerID="35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300" Nov 01 06:00:06.176572 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.176549 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt_odh-ai-gateway-infra(09037267-e69f-43e8-8ba5-d53b3dc179a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" Nov 01 06:00:06.177977 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.177959 4425 scope.go:122] "RemoveContainer" containerID="5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658" Nov 01 06:00:06.178184 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.178162 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm_odh-ai-gateway-infra(12762aaf-20be-4b5a-8bf3-3aab4cdfcc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" Nov 01 06:00:06.183376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.181569 4425 scope.go:122] "RemoveContainer" containerID="ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088" Nov 01 06:00:06.183376 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.181791 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6_odh-ai-gateway-infra(6e832485-6f1e-4a56-b8ce-16e25987de8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" Nov 01 06:00:06.185339 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:06.185320 4425 scope.go:122] "RemoveContainer" containerID="1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54" Nov 01 06:00:06.185571 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:06.185552 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4_odh-ai-gateway-infra(b0085083-fe55-40bf-a57f-cb544eb6be0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" Nov 01 06:00:07.773830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:07.773801 4425 scope.go:122] "RemoveContainer" containerID="0d68e235ee98c91a3936fc48294086efa5d10f1f2629611f6898e39b2dbaba45" Nov 01 06:00:07.785048 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:07.785023 4425 scope.go:122] "RemoveContainer" containerID="eab6d4090111d2deccb66b814f2742b8828d29835f12e984e04019e2486f8b64" Nov 01 06:00:07.797489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:07.797406 4425 scope.go:122] "RemoveContainer" containerID="489ee879a0202d67cdd805b1d4dffdac8156e9525549831af3dab192d291af84" Nov 01 06:00:14.971493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:14.971457 4425 scope.go:122] "RemoveContainer" containerID="97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556" Nov 01 06:00:15.966990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:15.966958 4425 scope.go:122] "RemoveContainer" containerID="047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab" Nov 01 06:00:15.967576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:15.967551 4425 scope.go:122] "RemoveContainer" containerID="ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088" Nov 01 06:00:16.271958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:16.271927 4425 generic.go:356] "Generic (PLEG): container finished" podID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerID="456abb14e88b325f484dd056062d21fc313db9572ffacb535f81d3ce07c7a499" exitCode=60 Nov 01 06:00:16.272325 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:16.272001 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" event={"ID":"8da493b6-987b-4011-8ffe-42f1207d4a4a","Type":"ContainerDied","Data":"456abb14e88b325f484dd056062d21fc313db9572ffacb535f81d3ce07c7a499"} Nov 01 06:00:16.272325 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:16.272063 4425 scope.go:122] "RemoveContainer" containerID="97cddbfece285e6a44f8be1af7816249deaabc495eca200e7995e07e7b179556" Nov 01 06:00:16.272452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:16.272434 4425 scope.go:122] "RemoveContainer" containerID="456abb14e88b325f484dd056062d21fc313db9572ffacb535f81d3ce07c7a499" Nov 01 06:00:16.272849 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:16.272785 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p_odh-ai-gateway-infra(8da493b6-987b-4011-8ffe-42f1207d4a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" Nov 01 06:00:16.966688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:16.966657 4425 scope.go:122] "RemoveContainer" containerID="ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713" Nov 01 06:00:17.280645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.280608 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerStarted","Data":"9276451c676e551435f67423c8c56c9a59bfbec3106b1de5c2358b5cc2b3bec4"} Nov 01 06:00:17.282680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.282655 4425 generic.go:356] "Generic (PLEG): container finished" podID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerID="680a6fd3000ef74c23c54e60413eaafc61a32b2e903a5f8c6e6b1c4d24864fbd" exitCode=60 Nov 01 06:00:17.282823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.282725 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerDied","Data":"680a6fd3000ef74c23c54e60413eaafc61a32b2e903a5f8c6e6b1c4d24864fbd"} Nov 01 06:00:17.282823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.282785 4425 scope.go:122] "RemoveContainer" containerID="ba08298dbebe77a910625fafc8046a283af04285eb2e3b06872c7f48ffae9088" Nov 01 06:00:17.283150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.283129 4425 scope.go:122] "RemoveContainer" containerID="680a6fd3000ef74c23c54e60413eaafc61a32b2e903a5f8c6e6b1c4d24864fbd" Nov 01 06:00:17.283491 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:17.283465 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6_odh-ai-gateway-infra(6e832485-6f1e-4a56-b8ce-16e25987de8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" Nov 01 06:00:17.284991 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.284973 4425 generic.go:356] "Generic (PLEG): container finished" podID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerID="f5a7b3273857b759947ba90c00d6046fda51b632e406198f7ee2db28a310dcb2" exitCode=60 Nov 01 06:00:17.285079 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.285024 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerDied","Data":"f5a7b3273857b759947ba90c00d6046fda51b632e406198f7ee2db28a310dcb2"} Nov 01 06:00:17.285409 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.285390 4425 scope.go:122] "RemoveContainer" containerID="f5a7b3273857b759947ba90c00d6046fda51b632e406198f7ee2db28a310dcb2" Nov 01 06:00:17.285664 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:17.285642 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl_odh-ai-gateway-infra(525dc9a9-3201-4e77-be83-2d33062a6a9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" Nov 01 06:00:17.296131 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.296110 4425 scope.go:122] "RemoveContainer" containerID="047e4a1c18b4c61b79c178030d749c160feb28dec8c6676d8f39d5dfda67d4ab" Nov 01 06:00:17.400420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.400386 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:17.560821 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.560793 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:17.575548 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.575520 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\") pod \"8da493b6-987b-4011-8ffe-42f1207d4a4a\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " Nov 01 06:00:17.575704 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.575632 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\" (UniqueName: \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\") pod \"8da493b6-987b-4011-8ffe-42f1207d4a4a\" (UID: \"8da493b6-987b-4011-8ffe-42f1207d4a4a\") " Nov 01 06:00:17.575962 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.575936 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca" pod "8da493b6-987b-4011-8ffe-42f1207d4a4a" (UID: "8da493b6-987b-4011-8ffe-42f1207d4a4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:17.577890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.577864 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q" pod "8da493b6-987b-4011-8ffe-42f1207d4a4a" (UID: "8da493b6-987b-4011-8ffe-42f1207d4a4a"). InnerVolumeSpecName "kube-api-access-zl42q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:17.677724 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.677682 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da493b6-987b-4011-8ffe-42f1207d4a4a-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:17.677724 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.677714 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl42q\" (UniqueName: \"kubernetes.io/projected/8da493b6-987b-4011-8ffe-42f1207d4a4a-kube-api-access-zl42q\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:17.966591 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.966559 4425 scope.go:122] "RemoveContainer" containerID="5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658" Nov 01 06:00:17.966760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.966698 4425 scope.go:122] "RemoveContainer" containerID="35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300" Nov 01 06:00:17.966903 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:17.966882 4425 scope.go:122] "RemoveContainer" containerID="1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54" Nov 01 06:00:18.296127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.296038 4425 generic.go:356] "Generic (PLEG): container finished" podID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerID="9276451c676e551435f67423c8c56c9a59bfbec3106b1de5c2358b5cc2b3bec4" exitCode=60 Nov 01 06:00:18.296127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.296112 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerDied","Data":"9276451c676e551435f67423c8c56c9a59bfbec3106b1de5c2358b5cc2b3bec4"} Nov 01 06:00:18.296691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.296200 4425 scope.go:122] "RemoveContainer" containerID="ef5a7fcba93ecdea97fe5e6b9e3aeffe5fb88e91875231630fc85a466aab5713" Nov 01 06:00:18.296691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.296628 4425 scope.go:122] "RemoveContainer" containerID="9276451c676e551435f67423c8c56c9a59bfbec3106b1de5c2358b5cc2b3bec4" Nov 01 06:00:18.296988 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:18.296964 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd_odh-ai-gateway-infra(a69944cd-44b1-4307-9680-76a3f62b86c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" Nov 01 06:00:18.298617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.298582 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerStarted","Data":"e9821c8b31f1584d0e75cd219b8780e0cea0040fcd7c9e6f5f4d00fe4eec7fed"} Nov 01 06:00:18.303695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.303669 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" event={"ID":"8da493b6-987b-4011-8ffe-42f1207d4a4a","Type":"ContainerDied","Data":"8be5f29e424450aa0e53fa3bbc3384952b2e598fa6a8b7498880ec7cb88e8bed"} Nov 01 06:00:18.303846 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.303696 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p" Nov 01 06:00:18.313507 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.313490 4425 scope.go:122] "RemoveContainer" containerID="456abb14e88b325f484dd056062d21fc313db9572ffacb535f81d3ce07c7a499" Nov 01 06:00:18.412684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.412643 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:18.542275 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.542224 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:18.548153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.548064 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c6e99c-29891880-mdj5p"] Nov 01 06:00:18.551704 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.551682 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:18.672892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.672862 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:18.720138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.720108 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:18.792386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.792358 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\" (UniqueName: \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\") pod \"6e832485-6f1e-4a56-b8ce-16e25987de8c\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " Nov 01 06:00:18.792575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.792435 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\") pod \"6e832485-6f1e-4a56-b8ce-16e25987de8c\" (UID: \"6e832485-6f1e-4a56-b8ce-16e25987de8c\") " Nov 01 06:00:18.793120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.793093 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca" pod "6e832485-6f1e-4a56-b8ce-16e25987de8c" (UID: "6e832485-6f1e-4a56-b8ce-16e25987de8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:18.795423 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.795392 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q" pod "6e832485-6f1e-4a56-b8ce-16e25987de8c" (UID: "6e832485-6f1e-4a56-b8ce-16e25987de8c"). InnerVolumeSpecName "kube-api-access-h4n7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:18.847954 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.847586 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:18.894049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894017 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\" (UniqueName: \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\") pod \"525dc9a9-3201-4e77-be83-2d33062a6a9d\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " Nov 01 06:00:18.894221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894205 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\") pod \"525dc9a9-3201-4e77-be83-2d33062a6a9d\" (UID: \"525dc9a9-3201-4e77-be83-2d33062a6a9d\") " Nov 01 06:00:18.894624 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894596 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca" pod "525dc9a9-3201-4e77-be83-2d33062a6a9d" (UID: "525dc9a9-3201-4e77-be83-2d33062a6a9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:18.894770 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894753 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e832485-6f1e-4a56-b8ce-16e25987de8c-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:18.894832 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894772 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525dc9a9-3201-4e77-be83-2d33062a6a9d-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:18.894832 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.894782 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4n7q\" (UniqueName: \"kubernetes.io/projected/6e832485-6f1e-4a56-b8ce-16e25987de8c-kube-api-access-h4n7q\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:18.896990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.896962 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k" pod "525dc9a9-3201-4e77-be83-2d33062a6a9d" (UID: "525dc9a9-3201-4e77-be83-2d33062a6a9d"). InnerVolumeSpecName "kube-api-access-f5p8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:18.966399 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.966360 4425 scope.go:122] "RemoveContainer" containerID="fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22" Nov 01 06:00:18.966571 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.966456 4425 scope.go:122] "RemoveContainer" containerID="c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715" Nov 01 06:00:18.974778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.974750 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" path="/var/lib/kubelet/pods/8da493b6-987b-4011-8ffe-42f1207d4a4a/volumes" Nov 01 06:00:18.996187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:18.996146 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5p8k\" (UniqueName: \"kubernetes.io/projected/525dc9a9-3201-4e77-be83-2d33062a6a9d-kube-api-access-f5p8k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:19.312271 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.312226 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" Nov 01 06:00:19.312783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.312265 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl" event={"ID":"525dc9a9-3201-4e77-be83-2d33062a6a9d","Type":"ContainerDied","Data":"606cc8aa45de523101c366863eb706dbe3bdeb251995c23802e2870d04a83913"} Nov 01 06:00:19.312783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.312353 4425 scope.go:122] "RemoveContainer" containerID="f5a7b3273857b759947ba90c00d6046fda51b632e406198f7ee2db28a310dcb2" Nov 01 06:00:19.317968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.317945 4425 generic.go:356] "Generic (PLEG): container finished" podID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerID="247958ee1a3b2ec0356c90c58bb870b50bbc8470c2aece65feee635b5834cb13" exitCode=60 Nov 01 06:00:19.318064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.318023 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerDied","Data":"247958ee1a3b2ec0356c90c58bb870b50bbc8470c2aece65feee635b5834cb13"} Nov 01 06:00:19.318459 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.318439 4425 scope.go:122] "RemoveContainer" containerID="247958ee1a3b2ec0356c90c58bb870b50bbc8470c2aece65feee635b5834cb13" Nov 01 06:00:19.318800 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:19.318775 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt_odh-ai-gateway-infra(09037267-e69f-43e8-8ba5-d53b3dc179a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" Nov 01 06:00:19.320224 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.320203 4425 generic.go:356] "Generic (PLEG): container finished" podID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerID="e9821c8b31f1584d0e75cd219b8780e0cea0040fcd7c9e6f5f4d00fe4eec7fed" exitCode=60 Nov 01 06:00:19.320357 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.320280 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerDied","Data":"e9821c8b31f1584d0e75cd219b8780e0cea0040fcd7c9e6f5f4d00fe4eec7fed"} Nov 01 06:00:19.320694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.320676 4425 scope.go:122] "RemoveContainer" containerID="e9821c8b31f1584d0e75cd219b8780e0cea0040fcd7c9e6f5f4d00fe4eec7fed" Nov 01 06:00:19.321008 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:19.320980 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm_odh-ai-gateway-infra(12762aaf-20be-4b5a-8bf3-3aab4cdfcc69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" Nov 01 06:00:19.321763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.321747 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" Nov 01 06:00:19.321870 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.321783 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6" event={"ID":"6e832485-6f1e-4a56-b8ce-16e25987de8c","Type":"ContainerDied","Data":"82ebb19b78cb6226e817e73b13f5740483ba558c552343bf3de516323be29daf"} Nov 01 06:00:19.324031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.324008 4425 generic.go:356] "Generic (PLEG): container finished" podID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerID="85feb935b6369f673b13f2bfd99f115f3268b41134858dadb528b6979338e366" exitCode=60 Nov 01 06:00:19.324120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.324071 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerDied","Data":"85feb935b6369f673b13f2bfd99f115f3268b41134858dadb528b6979338e366"} Nov 01 06:00:19.324457 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.324441 4425 scope.go:122] "RemoveContainer" containerID="35bbde7db347bb65a9fb156299eec3240b12a04b67076615c002308c7481e300" Nov 01 06:00:19.324515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.324500 4425 scope.go:122] "RemoveContainer" containerID="85feb935b6369f673b13f2bfd99f115f3268b41134858dadb528b6979338e366" Nov 01 06:00:19.324768 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:19.324748 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4_odh-ai-gateway-infra(b0085083-fe55-40bf-a57f-cb544eb6be0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" Nov 01 06:00:19.337979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.337958 4425 scope.go:122] "RemoveContainer" containerID="5b9a2c876689902241db1de0efc75c19cd23bceb2b4988a221ee87f83c0cd658" Nov 01 06:00:19.352287 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.352244 4425 scope.go:122] "RemoveContainer" containerID="680a6fd3000ef74c23c54e60413eaafc61a32b2e903a5f8c6e6b1c4d24864fbd" Nov 01 06:00:19.363589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.363336 4425 scope.go:122] "RemoveContainer" containerID="1e1f9f5a75bffc153c5635a7047fa4274f6aebc4f7cba7e12e732c085da7ac54" Nov 01 06:00:19.470029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.470001 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:19.481197 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.481170 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:19.484913 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.484890 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-098e4d-29891880-6tmh6"] Nov 01 06:00:19.503653 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.503627 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\" (UniqueName: \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\") pod \"a69944cd-44b1-4307-9680-76a3f62b86c7\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " Nov 01 06:00:19.503789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.503751 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\") pod \"a69944cd-44b1-4307-9680-76a3f62b86c7\" (UID: \"a69944cd-44b1-4307-9680-76a3f62b86c7\") " Nov 01 06:00:19.504177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.504150 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca" pod "a69944cd-44b1-4307-9680-76a3f62b86c7" (UID: "a69944cd-44b1-4307-9680-76a3f62b86c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:19.505994 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.505971 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj" pod "a69944cd-44b1-4307-9680-76a3f62b86c7" (UID: "a69944cd-44b1-4307-9680-76a3f62b86c7"). InnerVolumeSpecName "kube-api-access-s2ppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:19.596090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.596061 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:19.599801 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.599776 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c73135-29891880-g2wrl"] Nov 01 06:00:19.605563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.605536 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69944cd-44b1-4307-9680-76a3f62b86c7-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:19.605683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.605568 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ppj\" (UniqueName: \"kubernetes.io/projected/a69944cd-44b1-4307-9680-76a3f62b86c7-kube-api-access-s2ppj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:19.671037 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:19.671005 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:20.339893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.339861 4425 generic.go:356] "Generic (PLEG): container finished" podID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerID="0efca4c6eaab3aebe3e7518e512897c92d781b69b902a34da5cd4cbd0f651fa2" exitCode=60 Nov 01 06:00:20.340418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.339934 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerDied","Data":"0efca4c6eaab3aebe3e7518e512897c92d781b69b902a34da5cd4cbd0f651fa2"} Nov 01 06:00:20.340418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.340019 4425 scope.go:122] "RemoveContainer" containerID="fa4e3bd6a21eb3ac1aafa6388f9813ebf756eaabaeb7b41a897b8efa874f5f22" Nov 01 06:00:20.340535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.340443 4425 scope.go:122] "RemoveContainer" containerID="0efca4c6eaab3aebe3e7518e512897c92d781b69b902a34da5cd4cbd0f651fa2" Nov 01 06:00:20.340814 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:20.340791 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7_odh-ai-gateway-infra(967255e3-b934-487e-a9a0-ff3b1756a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" Nov 01 06:00:20.343063 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.343040 4425 generic.go:356] "Generic (PLEG): container finished" podID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerID="ad285e12f1ff84bb9cdd9585e9a8e20bb6461353371ca977a76a8489bfe5a889" exitCode=60 Nov 01 06:00:20.343165 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.343116 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerDied","Data":"ad285e12f1ff84bb9cdd9585e9a8e20bb6461353371ca977a76a8489bfe5a889"} Nov 01 06:00:20.343609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.343579 4425 scope.go:122] "RemoveContainer" containerID="ad285e12f1ff84bb9cdd9585e9a8e20bb6461353371ca977a76a8489bfe5a889" Nov 01 06:00:20.343929 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:20.343905 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw_odh-ai-gateway-infra(e0b14236-a698-44d9-b884-4b74b20a7ddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" Nov 01 06:00:20.344611 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.344595 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" Nov 01 06:00:20.344701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.344611 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd" event={"ID":"a69944cd-44b1-4307-9680-76a3f62b86c7","Type":"ContainerDied","Data":"bcba5791ba627cf4b998b37ea12dee456c7d3771560a3c8373fac2142cec7c76"} Nov 01 06:00:20.354811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.354794 4425 scope.go:122] "RemoveContainer" containerID="c55e79500eb319c5104116804b66122e2cebdb69944b7b98739b86de46a71715" Nov 01 06:00:20.368504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.368482 4425 scope.go:122] "RemoveContainer" containerID="9276451c676e551435f67423c8c56c9a59bfbec3106b1de5c2358b5cc2b3bec4" Nov 01 06:00:20.521996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.521969 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:20.572504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.572470 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:20.575962 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.575937 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dee6aa-29891880-npppd"] Nov 01 06:00:20.618324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.618229 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\") pod \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " Nov 01 06:00:20.618324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.618304 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\" (UniqueName: \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\") pod \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\" (UID: \"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69\") " Nov 01 06:00:20.618649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.618626 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca" pod "12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" (UID: "12762aaf-20be-4b5a-8bf3-3aab4cdfcc69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:20.618926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.618892 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:20.620686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.620663 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h" pod "12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" (UID: "12762aaf-20be-4b5a-8bf3-3aab4cdfcc69"). InnerVolumeSpecName "kube-api-access-lkp8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:20.719861 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.719822 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkp8h\" (UniqueName: \"kubernetes.io/projected/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69-kube-api-access-lkp8h\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:20.839459 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.839423 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:20.977675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.977646 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" path="/var/lib/kubelet/pods/525dc9a9-3201-4e77-be83-2d33062a6a9d/volumes" Nov 01 06:00:20.978097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.978083 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" path="/var/lib/kubelet/pods/6e832485-6f1e-4a56-b8ce-16e25987de8c/volumes" Nov 01 06:00:20.978482 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:20.978469 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" path="/var/lib/kubelet/pods/a69944cd-44b1-4307-9680-76a3f62b86c7/volumes" Nov 01 06:00:21.001951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.001923 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:21.076009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.075972 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:21.131297 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.131236 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\") pod \"b0085083-fe55-40bf-a57f-cb544eb6be0d\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " Nov 01 06:00:21.131451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.131319 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\" (UniqueName: \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\") pod \"b0085083-fe55-40bf-a57f-cb544eb6be0d\" (UID: \"b0085083-fe55-40bf-a57f-cb544eb6be0d\") " Nov 01 06:00:21.131685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.131658 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca" pod "b0085083-fe55-40bf-a57f-cb544eb6be0d" (UID: "b0085083-fe55-40bf-a57f-cb544eb6be0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:21.132112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.132093 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0085083-fe55-40bf-a57f-cb544eb6be0d-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.133717 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.133689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49" pod "b0085083-fe55-40bf-a57f-cb544eb6be0d" (UID: "b0085083-fe55-40bf-a57f-cb544eb6be0d"). InnerVolumeSpecName "kube-api-access-rls49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:21.233284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.233231 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rls49\" (UniqueName: \"kubernetes.io/projected/b0085083-fe55-40bf-a57f-cb544eb6be0d-kube-api-access-rls49\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.233719 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.233691 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:21.334165 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.334134 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\" (UniqueName: \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\") pod \"09037267-e69f-43e8-8ba5-d53b3dc179a0\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " Nov 01 06:00:21.334424 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.334193 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\") pod \"09037267-e69f-43e8-8ba5-d53b3dc179a0\" (UID: \"09037267-e69f-43e8-8ba5-d53b3dc179a0\") " Nov 01 06:00:21.334643 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.334618 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca" pod "09037267-e69f-43e8-8ba5-d53b3dc179a0" (UID: "09037267-e69f-43e8-8ba5-d53b3dc179a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:21.334822 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.334806 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09037267-e69f-43e8-8ba5-d53b3dc179a0-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.336693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.336658 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7" pod "09037267-e69f-43e8-8ba5-d53b3dc179a0" (UID: "09037267-e69f-43e8-8ba5-d53b3dc179a0"). InnerVolumeSpecName "kube-api-access-zmvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:21.362596 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.362570 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" Nov 01 06:00:21.363064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.362574 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt" event={"ID":"09037267-e69f-43e8-8ba5-d53b3dc179a0","Type":"ContainerDied","Data":"10582decc235395ff0b6a51f2a4d315548ea787b594cc5a03b343ab84150b164"} Nov 01 06:00:21.363064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.362707 4425 scope.go:122] "RemoveContainer" containerID="247958ee1a3b2ec0356c90c58bb870b50bbc8470c2aece65feee635b5834cb13" Nov 01 06:00:21.364043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.364025 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" Nov 01 06:00:21.364143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.364058 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm" event={"ID":"12762aaf-20be-4b5a-8bf3-3aab4cdfcc69","Type":"ContainerDied","Data":"edfee5907b91b79bef09da4067cac1d7331e7528d51db3718cff0a8c03f7e975"} Nov 01 06:00:21.365701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.365683 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" Nov 01 06:00:21.365823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.365713 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4" event={"ID":"b0085083-fe55-40bf-a57f-cb544eb6be0d","Type":"ContainerDied","Data":"ab5db61975847311d3a5848edf8d16914d94b0b119891e279f47f2652c7ea14d"} Nov 01 06:00:21.374104 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.374087 4425 scope.go:122] "RemoveContainer" containerID="e9821c8b31f1584d0e75cd219b8780e0cea0040fcd7c9e6f5f4d00fe4eec7fed" Nov 01 06:00:21.383694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.383673 4425 scope.go:122] "RemoveContainer" containerID="85feb935b6369f673b13f2bfd99f115f3268b41134858dadb528b6979338e366" Nov 01 06:00:21.436212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.436180 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvh7\" (UniqueName: \"kubernetes.io/projected/09037267-e69f-43e8-8ba5-d53b3dc179a0-kube-api-access-zmvh7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.468238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.468157 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:21.522594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.522563 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:21.528382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.528355 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-afc555-29891880-k7nkt"] Nov 01 06:00:21.607618 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.607590 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:21.658878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.658846 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:21.697773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.697743 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1c4191-29891880-v87x4"] Nov 01 06:00:21.740423 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.740342 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\" (UniqueName: \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\") pod \"967255e3-b934-487e-a9a0-ff3b1756a98b\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " Nov 01 06:00:21.740590 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.740477 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\") pod \"967255e3-b934-487e-a9a0-ff3b1756a98b\" (UID: \"967255e3-b934-487e-a9a0-ff3b1756a98b\") " Nov 01 06:00:21.740856 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.740832 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca" pod "967255e3-b934-487e-a9a0-ff3b1756a98b" (UID: "967255e3-b934-487e-a9a0-ff3b1756a98b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:21.740989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.740973 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/967255e3-b934-487e-a9a0-ff3b1756a98b-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.742649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.742629 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7" pod "967255e3-b934-487e-a9a0-ff3b1756a98b" (UID: "967255e3-b934-487e-a9a0-ff3b1756a98b"). InnerVolumeSpecName "kube-api-access-zxnp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:21.808779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.808747 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:21.812363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.812334 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7d1ec3-29891880-fhkmm"] Nov 01 06:00:21.815739 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.815717 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:21.842693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.842656 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxnp7\" (UniqueName: \"kubernetes.io/projected/967255e3-b934-487e-a9a0-ff3b1756a98b-kube-api-access-zxnp7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:21.981164 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:21.981139 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:22.044148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.044062 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\" (UniqueName: \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\") pod \"e0b14236-a698-44d9-b884-4b74b20a7ddf\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " Nov 01 06:00:22.044148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.044144 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\") pod \"e0b14236-a698-44d9-b884-4b74b20a7ddf\" (UID: \"e0b14236-a698-44d9-b884-4b74b20a7ddf\") " Nov 01 06:00:22.044553 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.044528 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca" pod "e0b14236-a698-44d9-b884-4b74b20a7ddf" (UID: "e0b14236-a698-44d9-b884-4b74b20a7ddf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:22.044700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.044686 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b14236-a698-44d9-b884-4b74b20a7ddf-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:22.046737 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.046707 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8" pod "e0b14236-a698-44d9-b884-4b74b20a7ddf" (UID: "e0b14236-a698-44d9-b884-4b74b20a7ddf"). InnerVolumeSpecName "kube-api-access-2lkm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:22.058902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.058875 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj"] Nov 01 06:00:22.064921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.064900 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed68a12c-bssrj"] Nov 01 06:00:22.146195 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.146161 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lkm8\" (UniqueName: \"kubernetes.io/projected/e0b14236-a698-44d9-b884-4b74b20a7ddf-kube-api-access-2lkm8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:22.376830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.376797 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" Nov 01 06:00:22.377350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.376797 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7" event={"ID":"967255e3-b934-487e-a9a0-ff3b1756a98b","Type":"ContainerDied","Data":"c76a708517700bade037cd1f980e294360015603348219ad046738923ff8e62d"} Nov 01 06:00:22.377350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.376983 4425 scope.go:122] "RemoveContainer" containerID="0efca4c6eaab3aebe3e7518e512897c92d781b69b902a34da5cd4cbd0f651fa2" Nov 01 06:00:22.378530 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.378507 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" event={"ID":"e0b14236-a698-44d9-b884-4b74b20a7ddf","Type":"ContainerDied","Data":"40932c95a6a04671993a1ce763bdab571e360fa7560d79aff4596231f198f7d0"} Nov 01 06:00:22.378612 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.378543 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw" Nov 01 06:00:22.388139 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.388120 4425 scope.go:122] "RemoveContainer" containerID="ad285e12f1ff84bb9cdd9585e9a8e20bb6461353371ca977a76a8489bfe5a889" Nov 01 06:00:22.548454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.548423 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:22.551770 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.551745 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7a360a-29891880-xltn7"] Nov 01 06:00:22.662958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.662875 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:22.666215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.666188 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d96a66-29891880-bs5jw"] Nov 01 06:00:22.974023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.973944 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" path="/var/lib/kubelet/pods/09037267-e69f-43e8-8ba5-d53b3dc179a0/volumes" Nov 01 06:00:22.974396 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.974383 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" path="/var/lib/kubelet/pods/12762aaf-20be-4b5a-8bf3-3aab4cdfcc69/volumes" Nov 01 06:00:22.974759 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.974747 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" path="/var/lib/kubelet/pods/967255e3-b934-487e-a9a0-ff3b1756a98b/volumes" Nov 01 06:00:22.975433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.975418 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" path="/var/lib/kubelet/pods/b0085083-fe55-40bf-a57f-cb544eb6be0d/volumes" Nov 01 06:00:22.975783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.975771 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" path="/var/lib/kubelet/pods/e0b14236-a698-44d9-b884-4b74b20a7ddf/volumes" Nov 01 06:00:22.976158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:22.976146 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa50a9e-d3b3-44bc-bc28-762ac9d270e3" path="/var/lib/kubelet/pods/eaa50a9e-d3b3-44bc-bc28-762ac9d270e3/volumes" Nov 01 06:00:26.947876 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:26.947839 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:00:41.841738 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.841688 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2"] Nov 01 06:00:41.842392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842373 4425 cpu_manager.go:414] "containerMap: removing container" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842396 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842410 4425 cpu_manager.go:414] "containerMap: removing container" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842417 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842426 4425 cpu_manager.go:414] "containerMap: removing container" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842431 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842440 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842444 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.842452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842451 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842457 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842466 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842470 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842478 4425 cpu_manager.go:414] "containerMap: removing container" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842482 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842498 4425 cpu_manager.go:414] "containerMap: removing container" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842502 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842507 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842512 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842519 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842523 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842534 4425 cpu_manager.go:414] "containerMap: removing container" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842538 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842546 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842550 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842562 4425 cpu_manager.go:414] "containerMap: removing container" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842584 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842597 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842603 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842608 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842613 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842621 4425 cpu_manager.go:414] "containerMap: removing container" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842625 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842631 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842636 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842642 4425 cpu_manager.go:414] "containerMap: removing container" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842646 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842653 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842657 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842664 4425 cpu_manager.go:414] "containerMap: removing container" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842668 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842680 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.842703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842684 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842776 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842785 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842792 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842798 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842804 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842809 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842815 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842821 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842825 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842830 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842835 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842839 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842844 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842849 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842854 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="525dc9a9-3201-4e77-be83-2d33062a6a9d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842860 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0085083-fe55-40bf-a57f-cb544eb6be0d" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842864 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842869 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842875 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842880 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842885 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842892 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:41.844358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.842896 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:41.846587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.846570 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:41.902722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.902691 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2"] Nov 01 06:00:41.904421 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.904393 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:41.904544 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.904522 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbr7k\" (UniqueName: \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:41.904588 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.904572 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:41.948840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.948804 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:00:41.948840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.948834 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:00:41.950326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.950308 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:00:41.950326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:41.950323 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:00:42.006314 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.006280 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.006513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.006355 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbr7k\" (UniqueName: \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.006513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.006393 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.007054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.007032 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.008897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.008867 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.068161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.068119 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbr7k\" (UniqueName: \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\") pod \"maas-api-revoke-sub-de0882d3-dzms2\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.160791 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.160697 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:42.543979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:42.543946 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2"] Nov 01 06:00:42.544362 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:42.544342 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1742b46_1376_4463_9cba_23eda1ce82a8.slice/crio-bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1 WatchSource:0}: Error finding container bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1: Status 404 returned error can't find the container with id bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1 Nov 01 06:00:43.542314 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:43.542282 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" event={"ID":"f1742b46-1376-4463-9cba-23eda1ce82a8","Type":"ContainerStarted","Data":"aca2471b36ecfa021dc9cdfb7be7ae94566245cdfa109d69f4e0e0a0f8203640"} Nov 01 06:00:43.542665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:43.542350 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" event={"ID":"f1742b46-1376-4463-9cba-23eda1ce82a8","Type":"ContainerStarted","Data":"bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1"} Nov 01 06:00:43.659750 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:43.659696 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" podStartSLOduration=2.659681481 podStartE2EDuration="2.659681481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:43.659459526 +0000 UTC m=+3458.800048446" watchObservedRunningTime="2026-11-01 06:00:43.659681481 +0000 UTC m=+3458.800270399" Nov 01 06:00:44.550987 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:44.550948 4425 generic.go:356] "Generic (PLEG): container finished" podID="f1742b46-1376-4463-9cba-23eda1ce82a8" containerID="aca2471b36ecfa021dc9cdfb7be7ae94566245cdfa109d69f4e0e0a0f8203640" exitCode=0 Nov 01 06:00:44.551421 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:44.551033 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" event={"ID":"f1742b46-1376-4463-9cba-23eda1ce82a8","Type":"ContainerDied","Data":"aca2471b36ecfa021dc9cdfb7be7ae94566245cdfa109d69f4e0e0a0f8203640"} Nov 01 06:00:45.706050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.706019 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:45.754139 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.754104 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\" (UniqueName: \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\") pod \"f1742b46-1376-4463-9cba-23eda1ce82a8\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " Nov 01 06:00:45.754378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.754199 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\") pod \"f1742b46-1376-4463-9cba-23eda1ce82a8\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " Nov 01 06:00:45.754378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.754319 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\") pod \"f1742b46-1376-4463-9cba-23eda1ce82a8\" (UID: \"f1742b46-1376-4463-9cba-23eda1ce82a8\") " Nov 01 06:00:45.754561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.754536 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca" pod "f1742b46-1376-4463-9cba-23eda1ce82a8" (UID: "f1742b46-1376-4463-9cba-23eda1ce82a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:45.754928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.754904 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:45.756629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.756597 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca" pod "f1742b46-1376-4463-9cba-23eda1ce82a8" (UID: "f1742b46-1376-4463-9cba-23eda1ce82a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:00:45.756629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.756609 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k" pod "f1742b46-1376-4463-9cba-23eda1ce82a8" (UID: "f1742b46-1376-4463-9cba-23eda1ce82a8"). InnerVolumeSpecName "kube-api-access-zbr7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:45.856027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.855990 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742b46-1376-4463-9cba-23eda1ce82a8-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:45.856027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:45.856023 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbr7k\" (UniqueName: \"kubernetes.io/projected/f1742b46-1376-4463-9cba-23eda1ce82a8-kube-api-access-zbr7k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:46.567445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:46.567352 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" event={"ID":"f1742b46-1376-4463-9cba-23eda1ce82a8","Type":"ContainerDied","Data":"bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1"} Nov 01 06:00:46.567445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:46.567404 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda7d8db304fdcca5fabd5eba0ba6753c5bbed40981f9c4fb12d06de91ae0de1" Nov 01 06:00:46.567445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:46.567403 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2" Nov 01 06:00:52.590210 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590171 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27"] Nov 01 06:00:52.590814 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590801 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:52.590874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590815 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:52.590874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590827 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f1742b46-1376-4463-9cba-23eda1ce82a8" containerName="revoke-keys" Nov 01 06:00:52.590874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590834 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1742b46-1376-4463-9cba-23eda1ce82a8" containerName="revoke-keys" Nov 01 06:00:52.590874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590858 4425 cpu_manager.go:414] "containerMap: removing container" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:52.590874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590863 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12762aaf-20be-4b5a-8bf3-3aab4cdfcc69" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590877 4425 cpu_manager.go:414] "containerMap: removing container" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590882 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590889 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590893 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da493b6-987b-4011-8ffe-42f1207d4a4a" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590912 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590917 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b14236-a698-44d9-b884-4b74b20a7ddf" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590923 4425 cpu_manager.go:414] "containerMap: removing container" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.590927 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.591009 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="967255e3-b934-487e-a9a0-ff3b1756a98b" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.591018 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1742b46-1376-4463-9cba-23eda1ce82a8" containerName="revoke-keys" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.591028 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e832485-6f1e-4a56-b8ce-16e25987de8c" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.591036 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69944cd-44b1-4307-9680-76a3f62b86c7" containerName="cleanup" Nov 01 06:00:52.591050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.591041 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="09037267-e69f-43e8-8ba5-d53b3dc179a0" containerName="cleanup" Nov 01 06:00:52.594566 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.594547 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.651677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.651640 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27"] Nov 01 06:00:52.657577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.657327 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.657577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.657385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dqq\" (UniqueName: \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.657577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.657431 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.697982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.697951 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:00:52.698128 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.697996 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:00:52.698128 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.698021 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:00:52.698353 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.698332 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:00:52.758889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.758856 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.759095 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.758947 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dqq\" (UniqueName: \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.759095 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.758997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.759696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.759631 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.761482 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.761457 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.820825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.820795 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dqq\" (UniqueName: \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\") pod \"maas-api-revoke-sub-3dd18e3c-6kj27\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:52.908640 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:52.908559 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:53.289667 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:53.289637 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27"] Nov 01 06:00:53.290076 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:00:53.290054 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd000678f_900b_4cc3_9fbc_e3a66511241b.slice/crio-cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f WatchSource:0}: Error finding container cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f: Status 404 returned error can't find the container with id cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f Nov 01 06:00:53.623751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:53.623718 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" event={"ID":"d000678f-900b-4cc3-9fbc-e3a66511241b","Type":"ContainerStarted","Data":"d797e5c8b778377c7501f5f49e20129deaaba3f6c981721696ceedf4aca35f41"} Nov 01 06:00:53.624217 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:53.623772 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" event={"ID":"d000678f-900b-4cc3-9fbc-e3a66511241b","Type":"ContainerStarted","Data":"cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f"} Nov 01 06:00:53.743720 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:53.743655 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" podStartSLOduration=1.7436362939999999 podStartE2EDuration="1.743636294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:53.740982861 +0000 UTC m=+3468.881571820" watchObservedRunningTime="2026-11-01 06:00:53.743636294 +0000 UTC m=+3468.884225219" Nov 01 06:00:54.633170 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:54.633139 4425 generic.go:356] "Generic (PLEG): container finished" podID="d000678f-900b-4cc3-9fbc-e3a66511241b" containerID="d797e5c8b778377c7501f5f49e20129deaaba3f6c981721696ceedf4aca35f41" exitCode=0 Nov 01 06:00:54.633651 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:54.633231 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" event={"ID":"d000678f-900b-4cc3-9fbc-e3a66511241b","Type":"ContainerDied","Data":"d797e5c8b778377c7501f5f49e20129deaaba3f6c981721696ceedf4aca35f41"} Nov 01 06:00:55.793425 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.793399 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:55.898561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.898473 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\") pod \"d000678f-900b-4cc3-9fbc-e3a66511241b\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " Nov 01 06:00:55.898561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.898561 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\") pod \"d000678f-900b-4cc3-9fbc-e3a66511241b\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " Nov 01 06:00:55.898798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.898671 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\" (UniqueName: \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\") pod \"d000678f-900b-4cc3-9fbc-e3a66511241b\" (UID: \"d000678f-900b-4cc3-9fbc-e3a66511241b\") " Nov 01 06:00:55.898941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.898916 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca" pod "d000678f-900b-4cc3-9fbc-e3a66511241b" (UID: "d000678f-900b-4cc3-9fbc-e3a66511241b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:00:55.899117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.899093 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:55.900804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.900781 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca" pod "d000678f-900b-4cc3-9fbc-e3a66511241b" (UID: "d000678f-900b-4cc3-9fbc-e3a66511241b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:00:55.901153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:55.901130 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq" pod "d000678f-900b-4cc3-9fbc-e3a66511241b" (UID: "d000678f-900b-4cc3-9fbc-e3a66511241b"). InnerVolumeSpecName "kube-api-access-d4dqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:00:56.000744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:56.000695 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d000678f-900b-4cc3-9fbc-e3a66511241b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:56.000744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:56.000740 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dqq\" (UniqueName: \"kubernetes.io/projected/d000678f-900b-4cc3-9fbc-e3a66511241b-kube-api-access-d4dqq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:00:56.649091 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:56.649064 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" Nov 01 06:00:56.649317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:56.649113 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27" event={"ID":"d000678f-900b-4cc3-9fbc-e3a66511241b","Type":"ContainerDied","Data":"cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f"} Nov 01 06:00:56.649317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:00:56.649165 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6d3b24a520b058c6aa8b4f53bc1101903dd9fe49c32148aefdac029477d23f" Nov 01 06:00:56.948491 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:00:56.948403 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:01:19.397171 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.397138 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn"] Nov 01 06:01:19.398074 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.397802 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d000678f-900b-4cc3-9fbc-e3a66511241b" containerName="revoke-keys" Nov 01 06:01:19.398074 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.397816 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d000678f-900b-4cc3-9fbc-e3a66511241b" containerName="revoke-keys" Nov 01 06:01:19.398074 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.397934 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d000678f-900b-4cc3-9fbc-e3a66511241b" containerName="revoke-keys" Nov 01 06:01:19.401603 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.401579 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.458049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.458013 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn"] Nov 01 06:01:19.504046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504013 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:01:19.504267 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504059 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:01:19.504267 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504064 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:01:19.504398 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504288 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7xr\" (UniqueName: \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.504446 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504429 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.504487 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.504457 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.505300 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.505286 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:01:19.605863 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.605822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7xr\" (UniqueName: \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.606047 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.605944 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.606047 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.605981 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.606868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.606808 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.608643 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.608607 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.668514 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.668444 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7xr\" (UniqueName: \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\") pod \"maas-api-revoke-sub-e238df4a-m7lbn\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:19.715840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:19.715815 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:20.094500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:20.094468 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn"] Nov 01 06:01:20.094639 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:01:20.094615 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268f6f37_4e71_4537_a5a7_66ad18986f25.slice/crio-919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b WatchSource:0}: Error finding container 919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b: Status 404 returned error can't find the container with id 919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b Nov 01 06:01:20.835855 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:20.835822 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" event={"ID":"268f6f37-4e71-4537-a5a7-66ad18986f25","Type":"ContainerStarted","Data":"6d0e1504366144feb866607fdedac733de69254f63966c31df22c22bc6d086dd"} Nov 01 06:01:20.836306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:20.835869 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" event={"ID":"268f6f37-4e71-4537-a5a7-66ad18986f25","Type":"ContainerStarted","Data":"919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b"} Nov 01 06:01:20.954858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:20.954800 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" podStartSLOduration=1.9547862820000002 podStartE2EDuration="1.954786282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:20.952555341 +0000 UTC m=+3496.093144261" watchObservedRunningTime="2026-11-01 06:01:20.954786282 +0000 UTC m=+3496.095375256" Nov 01 06:01:21.845961 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:21.845923 4425 generic.go:356] "Generic (PLEG): container finished" podID="268f6f37-4e71-4537-a5a7-66ad18986f25" containerID="6d0e1504366144feb866607fdedac733de69254f63966c31df22c22bc6d086dd" exitCode=0 Nov 01 06:01:21.846471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:21.846006 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" event={"ID":"268f6f37-4e71-4537-a5a7-66ad18986f25","Type":"ContainerDied","Data":"6d0e1504366144feb866607fdedac733de69254f63966c31df22c22bc6d086dd"} Nov 01 06:01:23.003860 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.003834 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:23.149810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.149726 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\") pod \"268f6f37-4e71-4537-a5a7-66ad18986f25\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " Nov 01 06:01:23.149810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.149786 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\") pod \"268f6f37-4e71-4537-a5a7-66ad18986f25\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " Nov 01 06:01:23.150067 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.149926 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\" (UniqueName: \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\") pod \"268f6f37-4e71-4537-a5a7-66ad18986f25\" (UID: \"268f6f37-4e71-4537-a5a7-66ad18986f25\") " Nov 01 06:01:23.150298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.150247 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca" pod "268f6f37-4e71-4537-a5a7-66ad18986f25" (UID: "268f6f37-4e71-4537-a5a7-66ad18986f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:23.152401 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.152374 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr" pod "268f6f37-4e71-4537-a5a7-66ad18986f25" (UID: "268f6f37-4e71-4537-a5a7-66ad18986f25"). InnerVolumeSpecName "kube-api-access-pj7xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:23.152949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.152924 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca" pod "268f6f37-4e71-4537-a5a7-66ad18986f25" (UID: "268f6f37-4e71-4537-a5a7-66ad18986f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:01:23.252413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.252379 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7xr\" (UniqueName: \"kubernetes.io/projected/268f6f37-4e71-4537-a5a7-66ad18986f25-kube-api-access-pj7xr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:23.252413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.252410 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:23.252413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.252420 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/268f6f37-4e71-4537-a5a7-66ad18986f25-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:23.860394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.860363 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" Nov 01 06:01:23.860662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.860396 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn" event={"ID":"268f6f37-4e71-4537-a5a7-66ad18986f25","Type":"ContainerDied","Data":"919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b"} Nov 01 06:01:23.860662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:23.860471 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919c8c52ae61e24491528ebb0df3a98ddb4a6ef3743eb9890d647bdc204a344b" Nov 01 06:01:26.410948 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:26.410909 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 06:01:26.948439 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:26.948404 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:01:27.961574 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:27.961542 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 06:01:32.564208 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.564175 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw"] Nov 01 06:01:32.564815 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.564801 4425 cpu_manager.go:414] "containerMap: removing container" podUID="268f6f37-4e71-4537-a5a7-66ad18986f25" containerName="revoke-keys" Nov 01 06:01:32.564875 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.564816 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268f6f37-4e71-4537-a5a7-66ad18986f25" containerName="revoke-keys" Nov 01 06:01:32.564937 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.564928 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="268f6f37-4e71-4537-a5a7-66ad18986f25" containerName="revoke-keys" Nov 01 06:01:32.569033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.569016 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.625130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.625089 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw"] Nov 01 06:01:32.661930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.661898 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22nzw\" (UniqueName: \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.662138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.662089 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.662215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.662169 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.671564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.671523 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:01:32.671564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.671542 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:01:32.672837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.672818 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:01:32.672945 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.672857 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:01:32.764047 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.764011 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.764237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.764086 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.764237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.764211 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22nzw\" (UniqueName: \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.764926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.764829 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.766988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.766968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.826099 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.826020 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22nzw\" (UniqueName: \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\") pod \"maas-api-revoke-sub-a88d1a54-m9ssw\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.883216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.883178 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:32.920853 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:32.920824 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 06:01:33.262620 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:01:33.262589 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d2e15f_dfbd_452b_a875_945270cc73ae.slice/crio-4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3 WatchSource:0}: Error finding container 4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3: Status 404 returned error can't find the container with id 4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3 Nov 01 06:01:33.263589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:33.263568 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw"] Nov 01 06:01:33.932266 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:33.932214 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" event={"ID":"d8d2e15f-dfbd-452b-a875-945270cc73ae","Type":"ContainerStarted","Data":"dbe68f1bef48428375be7cc2ad5fa62327755cf491ce216fc6668a5e151a06d1"} Nov 01 06:01:33.932665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:33.932331 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" event={"ID":"d8d2e15f-dfbd-452b-a875-945270cc73ae","Type":"ContainerStarted","Data":"4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3"} Nov 01 06:01:34.055401 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:34.055343 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" podStartSLOduration=2.055328483 podStartE2EDuration="2.055328483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:34.051217287 +0000 UTC m=+3509.191806207" watchObservedRunningTime="2026-11-01 06:01:34.055328483 +0000 UTC m=+3509.195917402" Nov 01 06:01:34.941237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:34.941201 4425 generic.go:356] "Generic (PLEG): container finished" podID="d8d2e15f-dfbd-452b-a875-945270cc73ae" containerID="dbe68f1bef48428375be7cc2ad5fa62327755cf491ce216fc6668a5e151a06d1" exitCode=0 Nov 01 06:01:34.941642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:34.941241 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" event={"ID":"d8d2e15f-dfbd-452b-a875-945270cc73ae","Type":"ContainerDied","Data":"dbe68f1bef48428375be7cc2ad5fa62327755cf491ce216fc6668a5e151a06d1"} Nov 01 06:01:35.684382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:35.684350 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 06:01:35.793774 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:35.793741 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 06:01:36.179897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.179861 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:36.211280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.211181 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\") pod \"d8d2e15f-dfbd-452b-a875-945270cc73ae\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " Nov 01 06:01:36.211469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.211395 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\") pod \"d8d2e15f-dfbd-452b-a875-945270cc73ae\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " Nov 01 06:01:36.211693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.211665 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca" pod "d8d2e15f-dfbd-452b-a875-945270cc73ae" (UID: "d8d2e15f-dfbd-452b-a875-945270cc73ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:36.211876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.211855 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\" (UniqueName: \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\") pod \"d8d2e15f-dfbd-452b-a875-945270cc73ae\" (UID: \"d8d2e15f-dfbd-452b-a875-945270cc73ae\") " Nov 01 06:01:36.212604 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.212563 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:36.214714 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.214689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca" pod "d8d2e15f-dfbd-452b-a875-945270cc73ae" (UID: "d8d2e15f-dfbd-452b-a875-945270cc73ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:01:36.215517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.215490 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw" pod "d8d2e15f-dfbd-452b-a875-945270cc73ae" (UID: "d8d2e15f-dfbd-452b-a875-945270cc73ae"). InnerVolumeSpecName "kube-api-access-22nzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:36.313975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.313942 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22nzw\" (UniqueName: \"kubernetes.io/projected/d8d2e15f-dfbd-452b-a875-945270cc73ae-kube-api-access-22nzw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:36.313975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.313973 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8d2e15f-dfbd-452b-a875-945270cc73ae-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:36.956197 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.956161 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" event={"ID":"d8d2e15f-dfbd-452b-a875-945270cc73ae","Type":"ContainerDied","Data":"4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3"} Nov 01 06:01:36.956410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.956213 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4241d4de9b17a5e79816bf34afdb01f4d0a33266474b4f3160c61d36e57035f3" Nov 01 06:01:36.956410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:36.956286 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw" Nov 01 06:01:37.921319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:37.921274 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" podUID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" containerName="payload-processing" containerID="cri-o://e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c" gracePeriod=25 Nov 01 06:01:38.195953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.195927 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 06:01:38.237933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.237896 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\") pod \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " Nov 01 06:01:38.238117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.237983 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\") pod \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " Nov 01 06:01:38.238117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238038 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\") pod \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " Nov 01 06:01:38.238117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238066 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\") pod \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\" (UID: \"54bc8339-7c9f-4b0f-8dc9-059dcabfc531\") " Nov 01 06:01:38.238479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238444 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume" pod "54bc8339-7c9f-4b0f-8dc9-059dcabfc531" (UID: "54bc8339-7c9f-4b0f-8dc9-059dcabfc531"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:38.238479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238473 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume" pod "54bc8339-7c9f-4b0f-8dc9-059dcabfc531" (UID: "54bc8339-7c9f-4b0f-8dc9-059dcabfc531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:38.238657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238602 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:38.238657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.238615 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:38.240438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.240414 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8" pod "54bc8339-7c9f-4b0f-8dc9-059dcabfc531" (UID: "54bc8339-7c9f-4b0f-8dc9-059dcabfc531"). InnerVolumeSpecName "kube-api-access-tg8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:38.240505 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.240459 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials" pod "54bc8339-7c9f-4b0f-8dc9-059dcabfc531" (UID: "54bc8339-7c9f-4b0f-8dc9-059dcabfc531"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:38.339881 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.339848 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:38.339881 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.339878 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8d8\" (UniqueName: \"kubernetes.io/projected/54bc8339-7c9f-4b0f-8dc9-059dcabfc531-kube-api-access-tg8d8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:38.972447 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.972413 4425 generic.go:356] "Generic (PLEG): container finished" podID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" containerID="e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c" exitCode=0 Nov 01 06:01:38.972947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.972535 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" Nov 01 06:01:38.976078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.976053 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" event={"ID":"54bc8339-7c9f-4b0f-8dc9-059dcabfc531","Type":"ContainerDied","Data":"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c"} Nov 01 06:01:38.976225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.976103 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j" event={"ID":"54bc8339-7c9f-4b0f-8dc9-059dcabfc531","Type":"ContainerDied","Data":"ca901eec759b79e6a3b71f5b9479c68873aec0357905c95cb2f84954be137262"} Nov 01 06:01:38.976225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.976131 4425 scope.go:122] "RemoveContainer" containerID="e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c" Nov 01 06:01:38.987068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.987052 4425 scope.go:122] "RemoveContainer" containerID="e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c" Nov 01 06:01:38.987369 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:38.987348 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c\": container with ID starting with e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c not found: ID does not exist" containerID="e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c" Nov 01 06:01:38.987424 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:38.987379 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c"} err="failed to get container status \"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c\": rpc error: code = NotFound desc = could not find container \"e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c\": container with ID starting with e134135b94bcbfcf64a282763d14fd8a0abb823fd0a6d9e9a95aec28e305fc1c not found: ID does not exist" Nov 01 06:01:39.142420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:39.142390 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 06:01:39.193105 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:39.193055 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-098e4d/payload-processing-external-model-e2e-extmod-b-098e4d-788497m6j"] Nov 01 06:01:40.684659 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:40.684604 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" containerID="cri-o://751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9" gracePeriod=25 Nov 01 06:01:40.794474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:40.794408 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" containerID="cri-o://5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e" gracePeriod=25 Nov 01 06:01:40.974392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:40.974320 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" path="/var/lib/kubelet/pods/54bc8339-7c9f-4b0f-8dc9-059dcabfc531/volumes" Nov 01 06:01:41.432161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.432122 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" containerName="main" containerID="cri-o://8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987" gracePeriod=45 Nov 01 06:01:41.696953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.696889 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 06:01:41.779783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.779749 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.779985 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.779937 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.779985 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.779966 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\" (UniqueName: \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.780127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.779996 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.780127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.780127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780048 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\" (UniqueName: \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\") pod \"883a2f7b-c3c6-466b-89f1-215477871d47\" (UID: \"883a2f7b-c3c6-466b-89f1-215477871d47\") " Nov 01 06:01:41.780320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780236 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:41.780320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780307 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:41.780437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780323 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:41.780588 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780563 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.780588 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780588 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.780750 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.780602 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.782156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.782130 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:41.782432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.782417 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:01:41.782559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.782540 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f" pod "883a2f7b-c3c6-466b-89f1-215477871d47" (UID: "883a2f7b-c3c6-466b-89f1-215477871d47"). InnerVolumeSpecName "kube-api-access-24g5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:41.881706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.881670 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g5f\" (UniqueName: \"kubernetes.io/projected/883a2f7b-c3c6-466b-89f1-215477871d47-kube-api-access-24g5f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.881706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.881700 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/883a2f7b-c3c6-466b-89f1-215477871d47-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.881706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.881713 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/883a2f7b-c3c6-466b-89f1-215477871d47-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:41.994150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.994057 4425 generic.go:356] "Generic (PLEG): container finished" podID="883a2f7b-c3c6-466b-89f1-215477871d47" containerID="8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987" exitCode=0 Nov 01 06:01:41.994150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.994098 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" Nov 01 06:01:41.994150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.994144 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" event={"ID":"883a2f7b-c3c6-466b-89f1-215477871d47","Type":"ContainerDied","Data":"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987"} Nov 01 06:01:41.994448 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.994186 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw" event={"ID":"883a2f7b-c3c6-466b-89f1-215477871d47","Type":"ContainerDied","Data":"ffc3b9e3095ef270663717b753aa4e3fd8cf8ca0669f8723c2dc2efe6dc77b92"} Nov 01 06:01:41.994448 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:41.994208 4425 scope.go:122] "RemoveContainer" containerID="8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987" Nov 01 06:01:42.004914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.004891 4425 scope.go:122] "RemoveContainer" containerID="8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987" Nov 01 06:01:42.005239 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:42.005222 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987\": container with ID starting with 8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987 not found: ID does not exist" containerID="8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987" Nov 01 06:01:42.005316 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.005265 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987"} err="failed to get container status \"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987\": rpc error: code = NotFound desc = could not find container \"8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987\": container with ID starting with 8a775482656fd84470b4e2e425b50ccfbe2deace728c4462373b8c59572ae987 not found: ID does not exist" Nov 01 06:01:42.162013 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.161975 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 06:01:42.165497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.165475 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-afc555/rate-test-model-afc555-kserve-59d9d79fbf-zx4lw"] Nov 01 06:01:42.982680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.982626 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" podUID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" containerName="main" containerID="cri-o://ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822" gracePeriod=45 Nov 01 06:01:42.983111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:42.982904 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" path="/var/lib/kubelet/pods/883a2f7b-c3c6-466b-89f1-215477871d47/volumes" Nov 01 06:01:43.265406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.265376 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 06:01:43.303169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303130 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303196 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303234 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303290 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\" (UniqueName: \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303370 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\" (UniqueName: \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303493 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\") pod \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\" (UID: \"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718\") " Nov 01 06:01:43.303609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303501 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:43.303609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303551 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:43.303788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.303718 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:43.304169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.304079 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:43.304169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.304099 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:43.304169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.304134 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:43.305816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.305789 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:01:43.305958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.305936 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:01:43.306028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.306009 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn" pod "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" (UID: "811a9a71-fc8f-44d9-9a0b-d6ecf94e8718"). InnerVolumeSpecName "kube-api-access-cpvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:43.409168 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.409120 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvqn\" (UniqueName: \"kubernetes.io/projected/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-kube-api-access-cpvqn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:43.409168 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.409160 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:43.409168 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:43.409175 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:44.013432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.013392 4425 generic.go:356] "Generic (PLEG): container finished" podID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" containerID="ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822" exitCode=0 Nov 01 06:01:44.013889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.013440 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" event={"ID":"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718","Type":"ContainerDied","Data":"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822"} Nov 01 06:01:44.013889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.013470 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" Nov 01 06:01:44.013889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.013495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6" event={"ID":"811a9a71-fc8f-44d9-9a0b-d6ecf94e8718","Type":"ContainerDied","Data":"e78b0b2a8e200f31fd353404a94d44778adf1258e57706f11a0836247b95cfc5"} Nov 01 06:01:44.013889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.013518 4425 scope.go:122] "RemoveContainer" containerID="ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822" Nov 01 06:01:44.026082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.026063 4425 scope.go:122] "RemoveContainer" containerID="ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822" Nov 01 06:01:44.026389 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:44.026363 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822\": container with ID starting with ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822 not found: ID does not exist" containerID="ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822" Nov 01 06:01:44.026479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.026395 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822"} err="failed to get container status \"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822\": rpc error: code = NotFound desc = could not find container \"ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822\": container with ID starting with ef71d7e7e0a073e96e45a52767b14990eae71cc46a71b844ac1398fcb87ad822 not found: ID does not exist" Nov 01 06:01:44.181716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.181683 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 06:01:44.187966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.187939 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1c4191/rate-test-model-1c4191-kserve-67789c6677-7lgk6"] Nov 01 06:01:44.975450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.975414 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" path="/var/lib/kubelet/pods/811a9a71-fc8f-44d9-9a0b-d6ecf94e8718/volumes" Nov 01 06:01:44.978058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:44.978039 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 06:01:49.537554 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:49.537515 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:01:49.538008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:49.537516 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:01:49.978384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:49.978342 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" podUID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" containerName="payload-processing" containerID="cri-o://df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364" gracePeriod=25 Nov 01 06:01:50.239922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.239856 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 06:01:50.401126 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401088 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\") pod \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " Nov 01 06:01:50.401391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401156 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\") pod \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " Nov 01 06:01:50.401391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401360 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\") pod \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " Nov 01 06:01:50.401513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401486 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\") pod \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\" (UID: \"33eaea69-89fa-404f-8c45-e7a07aa05eb2\") " Nov 01 06:01:50.401594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401562 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume" pod "33eaea69-89fa-404f-8c45-e7a07aa05eb2" (UID: "33eaea69-89fa-404f-8c45-e7a07aa05eb2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:50.401886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.401862 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume" pod "33eaea69-89fa-404f-8c45-e7a07aa05eb2" (UID: "33eaea69-89fa-404f-8c45-e7a07aa05eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:01:50.402191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.402169 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:50.402276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.402201 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eaea69-89fa-404f-8c45-e7a07aa05eb2-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:50.403730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.403698 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials" pod "33eaea69-89fa-404f-8c45-e7a07aa05eb2" (UID: "33eaea69-89fa-404f-8c45-e7a07aa05eb2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:50.403730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.403716 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm" pod "33eaea69-89fa-404f-8c45-e7a07aa05eb2" (UID: "33eaea69-89fa-404f-8c45-e7a07aa05eb2"). InnerVolumeSpecName "kube-api-access-46vxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:01:50.503765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.503671 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vxm\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-kube-api-access-46vxm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:50.503765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:50.503707 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33eaea69-89fa-404f-8c45-e7a07aa05eb2-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:01:51.064136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.064102 4425 generic.go:356] "Generic (PLEG): container finished" podID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" containerID="df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364" exitCode=0 Nov 01 06:01:51.064610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.064150 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" Nov 01 06:01:51.064610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.064188 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" event={"ID":"33eaea69-89fa-404f-8c45-e7a07aa05eb2","Type":"ContainerDied","Data":"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364"} Nov 01 06:01:51.064610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.064247 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp" event={"ID":"33eaea69-89fa-404f-8c45-e7a07aa05eb2","Type":"ContainerDied","Data":"2c1f44b3c9044f350333fc02fffa8e6193c2c82e7cb89a9c6a783bff76ad0fe0"} Nov 01 06:01:51.064610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.064282 4425 scope.go:122] "RemoveContainer" containerID="df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364" Nov 01 06:01:51.074069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.074038 4425 scope.go:122] "RemoveContainer" containerID="df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364" Nov 01 06:01:51.074355 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:51.074336 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364\": container with ID starting with df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364 not found: ID does not exist" containerID="df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364" Nov 01 06:01:51.074426 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.074361 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364"} err="failed to get container status \"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364\": rpc error: code = NotFound desc = could not find container \"df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364\": container with ID starting with df7bda2d178a42f61c3c4d1a2721e8efeec338dc43bbac14863576c5ef91d364 not found: ID does not exist" Nov 01 06:01:51.232327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.232226 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 06:01:51.283514 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:51.283485 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d96a66/payload-processing-external-model-e2e-extmod-a-d96a66-85d76njtp"] Nov 01 06:01:52.978029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:52.977996 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" path="/var/lib/kubelet/pods/33eaea69-89fa-404f-8c45-e7a07aa05eb2/volumes" Nov 01 06:01:56.947583 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:01:56.947543 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:01:58.947620 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:58.947586 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 06:01:59.055151 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:59.055115 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 06:01:59.537406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:59.537367 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:01:59.538047 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:01:59.538012 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:00.965870 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:00.965837 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 06:02:01.036382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.036336 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\" (UniqueName: \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\") pod \"0bd8ec32-8959-434b-bc5e-17911908e4c1\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " Nov 01 06:02:01.036649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.036613 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\") pod \"0bd8ec32-8959-434b-bc5e-17911908e4c1\" (UID: \"0bd8ec32-8959-434b-bc5e-17911908e4c1\") " Nov 01 06:02:01.037997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.037951 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume" pod "0bd8ec32-8959-434b-bc5e-17911908e4c1" (UID: "0bd8ec32-8959-434b-bc5e-17911908e4c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:01.041106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.041076 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g" pod "0bd8ec32-8959-434b-bc5e-17911908e4c1" (UID: "0bd8ec32-8959-434b-bc5e-17911908e4c1"). InnerVolumeSpecName "kube-api-access-hpv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:01.063756 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.063733 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 06:02:01.137934 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.137904 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\" (UniqueName: \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\") pod \"b139b5f9-155f-44ce-8488-02c0dfd3530f\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " Nov 01 06:02:01.138121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.137997 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\") pod \"b139b5f9-155f-44ce-8488-02c0dfd3530f\" (UID: \"b139b5f9-155f-44ce-8488-02c0dfd3530f\") " Nov 01 06:02:01.138367 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.138353 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd8ec32-8959-434b-bc5e-17911908e4c1-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:01.138436 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.138368 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpv4g\" (UniqueName: \"kubernetes.io/projected/0bd8ec32-8959-434b-bc5e-17911908e4c1-kube-api-access-hpv4g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:01.138436 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.138402 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume" pod "b139b5f9-155f-44ce-8488-02c0dfd3530f" (UID: "b139b5f9-155f-44ce-8488-02c0dfd3530f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:01.140281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.140243 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh" pod "b139b5f9-155f-44ce-8488-02c0dfd3530f" (UID: "b139b5f9-155f-44ce-8488-02c0dfd3530f"). InnerVolumeSpecName "kube-api-access-m2mmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:01.163448 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.163414 4425 generic.go:356] "Generic (PLEG): container finished" podID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerID="5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e" exitCode=0 Nov 01 06:02:01.163587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.163458 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" Nov 01 06:02:01.163587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.163491 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" event={"ID":"b139b5f9-155f-44ce-8488-02c0dfd3530f","Type":"ContainerDied","Data":"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e"} Nov 01 06:02:01.163587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.163548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn" event={"ID":"b139b5f9-155f-44ce-8488-02c0dfd3530f","Type":"ContainerDied","Data":"f4438f56cc3bebfb065d8be7f8a7b922a3adf58b51be54a98ab0322d69e2f742"} Nov 01 06:02:01.163587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.163582 4425 scope.go:122] "RemoveContainer" containerID="5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e" Nov 01 06:02:01.165103 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.165076 4425 generic.go:356] "Generic (PLEG): container finished" podID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerID="751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9" exitCode=0 Nov 01 06:02:01.165239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.165134 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" Nov 01 06:02:01.165239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.165164 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" event={"ID":"0bd8ec32-8959-434b-bc5e-17911908e4c1","Type":"ContainerDied","Data":"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9"} Nov 01 06:02:01.165239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.165232 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx" event={"ID":"0bd8ec32-8959-434b-bc5e-17911908e4c1","Type":"ContainerDied","Data":"458c6f25624f9f8a2772212dfcad7023b1907eec75122f12b2caf6832ecc4e1c"} Nov 01 06:02:01.176771 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.176751 4425 scope.go:122] "RemoveContainer" containerID="5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e" Nov 01 06:02:01.177087 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:01.177070 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e\": container with ID starting with 5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e not found: ID does not exist" containerID="5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e" Nov 01 06:02:01.177149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.177101 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e"} err="failed to get container status \"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e\": rpc error: code = NotFound desc = could not find container \"5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e\": container with ID starting with 5bc7771d4db156c6674e99f1b5ab8f9427f7846f4dfbb8fd1669b16902a80b0e not found: ID does not exist" Nov 01 06:02:01.177149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.177122 4425 scope.go:122] "RemoveContainer" containerID="751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9" Nov 01 06:02:01.186685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.186666 4425 scope.go:122] "RemoveContainer" containerID="751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9" Nov 01 06:02:01.187029 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:01.186983 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9\": container with ID starting with 751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9 not found: ID does not exist" containerID="751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9" Nov 01 06:02:01.187029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.187010 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9"} err="failed to get container status \"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9\": rpc error: code = NotFound desc = could not find container \"751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9\": container with ID starting with 751884beafe8298a87335340068a40cf3c6a8e0a79825f1f90faf75dc99fb4b9 not found: ID does not exist" Nov 01 06:02:01.239292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.239182 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mmh\" (UniqueName: \"kubernetes.io/projected/b139b5f9-155f-44ce-8488-02c0dfd3530f-kube-api-access-m2mmh\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:01.239292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.239217 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b139b5f9-155f-44ce-8488-02c0dfd3530f-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:01.316991 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.316953 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 06:02:01.324516 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.324483 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c73135-7575bf697f-mstqx"] Nov 01 06:02:01.401834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.401784 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 06:02:01.434107 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.434066 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 06:02:01.438280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.438228 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c73135-78b44ccf5-884wn"] Nov 01 06:02:01.513067 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:01.512981 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 06:02:02.974091 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:02.974052 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" path="/var/lib/kubelet/pods/0bd8ec32-8959-434b-bc5e-17911908e4c1/volumes" Nov 01 06:02:02.974500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:02.974480 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" path="/var/lib/kubelet/pods/b139b5f9-155f-44ce-8488-02c0dfd3530f/volumes" Nov 01 06:02:03.948359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:03.948312 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" containerID="cri-o://9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf" gracePeriod=25 Nov 01 06:02:04.056037 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:04.055995 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" containerID="cri-o://e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e" gracePeriod=25 Nov 01 06:02:06.402409 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:06.402356 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" containerID="cri-o://20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605" gracePeriod=25 Nov 01 06:02:06.514039 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:06.513978 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" containerID="cri-o://b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0" gracePeriod=25 Nov 01 06:02:07.924755 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:07.924724 4425 scope.go:122] "RemoveContainer" containerID="622999c0d0e2088b501d705b1c0c8e6334d5a2dbb1cd261c7d22e5d009b4e0b4" Nov 01 06:02:09.458068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:09.458033 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:09.458068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:09.458055 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:14.821578 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:14.821533 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:15.828703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:15.828659 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:19.458747 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:19.458659 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:19.458747 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:19.458699 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:23.024103 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024072 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:02:23.024708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024693 4425 cpu_manager.go:414] "containerMap: removing container" podUID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" containerName="payload-processing" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024709 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" containerName="payload-processing" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024728 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024733 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024742 4425 cpu_manager.go:414] "containerMap: removing container" podUID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" containerName="main" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024747 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" containerName="main" Nov 01 06:02:23.024758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024758 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d8d2e15f-dfbd-452b-a875-945270cc73ae" containerName="revoke-keys" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024762 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d2e15f-dfbd-452b-a875-945270cc73ae" containerName="revoke-keys" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024771 4425 cpu_manager.go:414] "containerMap: removing container" podUID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" containerName="payload-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024775 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" containerName="payload-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024789 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024793 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024807 4425 cpu_manager.go:414] "containerMap: removing container" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" containerName="main" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024814 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" containerName="main" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024913 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b139b5f9-155f-44ce-8488-02c0dfd3530f" containerName="payload-pre-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024922 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="883a2f7b-c3c6-466b-89f1-215477871d47" containerName="main" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024933 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="33eaea69-89fa-404f-8c45-e7a07aa05eb2" containerName="payload-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024940 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd8ec32-8959-434b-bc5e-17911908e4c1" containerName="payload-processing" Nov 01 06:02:23.024943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024945 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="811a9a71-fc8f-44d9-9a0b-d6ecf94e8718" containerName="main" Nov 01 06:02:23.025380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024949 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d2e15f-dfbd-452b-a875-945270cc73ae" containerName="revoke-keys" Nov 01 06:02:23.025380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.024954 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bc8339-7c9f-4b0f-8dc9-059dcabfc531" containerName="payload-processing" Nov 01 06:02:23.028394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.028372 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.114701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114477 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.114701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114517 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.114701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114682 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114716 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114789 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114842 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxhx\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114876 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114933 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.115245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.114968 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.138269 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.138210 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:02:23.216557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216525 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxhx\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.216746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216570 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.216746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216610 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.216746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216634 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.216746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216687 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.216746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216710 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216797 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216913 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.216973 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.217098 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.217143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.217181 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.217425 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.217652 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.217625 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.219380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.219359 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.219504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.219486 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.278306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.278216 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.278478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.278380 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxhx\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-trkbg\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.341756 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.341723 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:23.782310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.778910 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:02:23.785164 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:02:23.785085 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c7bd23_7c17_4815_ae5d_e90de1e49ac8.slice/crio-1eed339a2a9a68715b428c62571ea635133d2f6fa644f4a9134f2664fcae8b1f WatchSource:0}: Error finding container 1eed339a2a9a68715b428c62571ea635133d2f6fa644f4a9134f2664fcae8b1f: Status 404 returned error can't find the container with id 1eed339a2a9a68715b428c62571ea635133d2f6fa644f4a9134f2664fcae8b1f Nov 01 06:02:23.789515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.789467 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:02:23.789632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.789571 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:02:23.789632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:23.789621 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:02:24.190186 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.190153 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 06:02:24.330313 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.330217 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\") pod \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " Nov 01 06:02:24.330483 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.330316 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\" (UniqueName: \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\") pod \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\" (UID: \"ddfabf84-d74e-41c4-bc9a-0c09e88c512b\") " Nov 01 06:02:24.330681 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.330654 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume" pod "ddfabf84-d74e-41c4-bc9a-0c09e88c512b" (UID: "ddfabf84-d74e-41c4-bc9a-0c09e88c512b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:24.330883 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.330863 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:24.333695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.333668 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l" pod "ddfabf84-d74e-41c4-bc9a-0c09e88c512b" (UID: "ddfabf84-d74e-41c4-bc9a-0c09e88c512b"). InnerVolumeSpecName "kube-api-access-p4m5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:24.335389 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.335363 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 06:02:24.345733 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.345696 4425 generic.go:356] "Generic (PLEG): container finished" podID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerID="e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e" exitCode=0 Nov 01 06:02:24.345880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.345859 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" Nov 01 06:02:24.346017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.345881 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" event={"ID":"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1","Type":"ContainerDied","Data":"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e"} Nov 01 06:02:24.346017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.345945 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd" event={"ID":"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1","Type":"ContainerDied","Data":"103c5e769ddd28a07b3bb7275a3e77eace83eed15f7f04558f609f1c796e3b5b"} Nov 01 06:02:24.346017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.345980 4425 scope.go:122] "RemoveContainer" containerID="e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e" Nov 01 06:02:24.348598 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.348569 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" event={"ID":"86c7bd23-7c17-4815-ae5d-e90de1e49ac8","Type":"ContainerStarted","Data":"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32"} Nov 01 06:02:24.348718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.348627 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" event={"ID":"86c7bd23-7c17-4815-ae5d-e90de1e49ac8","Type":"ContainerStarted","Data":"1eed339a2a9a68715b428c62571ea635133d2f6fa644f4a9134f2664fcae8b1f"} Nov 01 06:02:24.351239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.351213 4425 generic.go:356] "Generic (PLEG): container finished" podID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerID="9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf" exitCode=0 Nov 01 06:02:24.351378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.351295 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" Nov 01 06:02:24.351378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.351332 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" event={"ID":"ddfabf84-d74e-41c4-bc9a-0c09e88c512b","Type":"ContainerDied","Data":"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf"} Nov 01 06:02:24.351378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.351376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z" event={"ID":"ddfabf84-d74e-41c4-bc9a-0c09e88c512b","Type":"ContainerDied","Data":"49cfed330ae59b2565c34a34d1b17b7fedb1f4dd04067fbf9518ecb32587383b"} Nov 01 06:02:24.364458 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.364173 4425 scope.go:122] "RemoveContainer" containerID="e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e" Nov 01 06:02:24.364784 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:24.364683 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e\": container with ID starting with e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e not found: ID does not exist" containerID="e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e" Nov 01 06:02:24.364784 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.364713 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e"} err="failed to get container status \"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e\": rpc error: code = NotFound desc = could not find container \"e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e\": container with ID starting with e70c6d14310c194ef434b2e2f80dfbb1de20a9ddf7eb00d29ea0f047ab7e839e not found: ID does not exist" Nov 01 06:02:24.364784 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.364730 4425 scope.go:122] "RemoveContainer" containerID="9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf" Nov 01 06:02:24.388594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.388551 4425 scope.go:122] "RemoveContainer" containerID="9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf" Nov 01 06:02:24.389473 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:24.389442 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf\": container with ID starting with 9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf not found: ID does not exist" containerID="9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf" Nov 01 06:02:24.389707 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.389679 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf"} err="failed to get container status \"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf\": rpc error: code = NotFound desc = could not find container \"9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf\": container with ID starting with 9af4e9e8add336d27ca68c698e46a8ca1c567774c89050f87a09c5152cc181cf not found: ID does not exist" Nov 01 06:02:24.432333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.432304 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\" (UniqueName: \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\") pod \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " Nov 01 06:02:24.432484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.432466 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\") pod \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\" (UID: \"1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1\") " Nov 01 06:02:24.432748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.432733 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4m5l\" (UniqueName: \"kubernetes.io/projected/ddfabf84-d74e-41c4-bc9a-0c09e88c512b-kube-api-access-p4m5l\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:24.432843 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.432821 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume" pod "1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" (UID: "1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:24.434752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.434718 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll" pod "1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" (UID: "1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1"). InnerVolumeSpecName "kube-api-access-v4jll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:24.534504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.534470 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:24.534504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.534507 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4jll\" (UniqueName: \"kubernetes.io/projected/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1-kube-api-access-v4jll\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:24.571420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.571385 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 06:02:24.574690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.574670 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dee6aa-cddd945f5-lmc2z"] Nov 01 06:02:24.696190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.693528 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podStartSLOduration=2.693513031 podStartE2EDuration="2.693513031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:24.691988711 +0000 UTC m=+3559.832577631" watchObservedRunningTime="2026-11-01 06:02:24.693513031 +0000 UTC m=+3559.834101950" Nov 01 06:02:24.839176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.839133 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:24.880175 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.880132 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 06:02:24.883627 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:24.883597 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dee6aa-864f9cfc9f-ljqpd"] Nov 01 06:02:25.002519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.002417 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" path="/var/lib/kubelet/pods/1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1/volumes" Nov 01 06:02:25.002872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.002855 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" path="/var/lib/kubelet/pods/ddfabf84-d74e-41c4-bc9a-0c09e88c512b/volumes" Nov 01 06:02:25.345648 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.345516 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:25.347445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.347072 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:25.347445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.347142 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:25.828029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:25.827991 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:26.347123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.346648 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:26.347123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.346711 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:26.738213 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.738171 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 06:02:26.840657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.840630 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 06:02:26.884357 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.884239 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\") pod \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " Nov 01 06:02:26.884623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.884602 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\" (UniqueName: \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\") pod \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\" (UID: \"b2de60f3-08d3-4146-adb6-608eb8cbcbc6\") " Nov 01 06:02:26.884711 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.884646 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume" pod "b2de60f3-08d3-4146-adb6-608eb8cbcbc6" (UID: "b2de60f3-08d3-4146-adb6-608eb8cbcbc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:26.885066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.885045 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:26.887046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.887017 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5" pod "b2de60f3-08d3-4146-adb6-608eb8cbcbc6" (UID: "b2de60f3-08d3-4146-adb6-608eb8cbcbc6"). InnerVolumeSpecName "kube-api-access-wfzn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:26.979456 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:26.979418 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:02:26.985509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.985477 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\" (UniqueName: \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\") pod \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " Nov 01 06:02:26.985687 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.985548 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\") pod \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\" (UID: \"55c746c6-d0ec-4333-8bdd-ccb57adfd329\") " Nov 01 06:02:26.986004 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.985974 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume" pod "55c746c6-d0ec-4333-8bdd-ccb57adfd329" (UID: "55c746c6-d0ec-4333-8bdd-ccb57adfd329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:26.986191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.986174 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfzn5\" (UniqueName: \"kubernetes.io/projected/b2de60f3-08d3-4146-adb6-608eb8cbcbc6-kube-api-access-wfzn5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:26.986331 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.986195 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c746c6-d0ec-4333-8bdd-ccb57adfd329-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:26.989649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:26.989625 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh" pod "55c746c6-d0ec-4333-8bdd-ccb57adfd329" (UID: "55c746c6-d0ec-4333-8bdd-ccb57adfd329"). InnerVolumeSpecName "kube-api-access-2szzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:27.087278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.087213 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szzh\" (UniqueName: \"kubernetes.io/projected/55c746c6-d0ec-4333-8bdd-ccb57adfd329-kube-api-access-2szzh\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:27.361280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.353974 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:27.361280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.354048 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:27.387386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.387215 4425 generic.go:356] "Generic (PLEG): container finished" podID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerID="20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605" exitCode=0 Nov 01 06:02:27.387386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.387289 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" Nov 01 06:02:27.387386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.387299 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" event={"ID":"b2de60f3-08d3-4146-adb6-608eb8cbcbc6","Type":"ContainerDied","Data":"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605"} Nov 01 06:02:27.387386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.387355 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf" event={"ID":"b2de60f3-08d3-4146-adb6-608eb8cbcbc6","Type":"ContainerDied","Data":"fce9060c791042c9f7ec7c214fd9b12808a964075cc8b24f6363a057efa4e6fc"} Nov 01 06:02:27.387785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.387400 4425 scope.go:122] "RemoveContainer" containerID="20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605" Nov 01 06:02:27.389474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.389380 4425 generic.go:356] "Generic (PLEG): container finished" podID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerID="b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0" exitCode=0 Nov 01 06:02:27.389474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.389432 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" Nov 01 06:02:27.389655 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.389425 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" event={"ID":"55c746c6-d0ec-4333-8bdd-ccb57adfd329","Type":"ContainerDied","Data":"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0"} Nov 01 06:02:27.389655 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.389545 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd" event={"ID":"55c746c6-d0ec-4333-8bdd-ccb57adfd329","Type":"ContainerDied","Data":"68d3c2ed3374e18a710ebd0d367201c6e242d8285959dd2343d0f02a636fac7d"} Nov 01 06:02:27.399141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.399125 4425 scope.go:122] "RemoveContainer" containerID="20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605" Nov 01 06:02:27.399445 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:27.399427 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605\": container with ID starting with 20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605 not found: ID does not exist" containerID="20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605" Nov 01 06:02:27.399489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.399453 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605"} err="failed to get container status \"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605\": rpc error: code = NotFound desc = could not find container \"20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605\": container with ID starting with 20cecf660efe2edd2e34c98b0545669cf05fcbabce14476da4039c088ae40605 not found: ID does not exist" Nov 01 06:02:27.399489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.399471 4425 scope.go:122] "RemoveContainer" containerID="b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0" Nov 01 06:02:27.409090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.409063 4425 scope.go:122] "RemoveContainer" containerID="b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0" Nov 01 06:02:27.409424 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:27.409406 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0\": container with ID starting with b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0 not found: ID does not exist" containerID="b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0" Nov 01 06:02:27.409499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.409430 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0"} err="failed to get container status \"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0\": rpc error: code = NotFound desc = could not find container \"b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0\": container with ID starting with b98113c14ef8c1ba7a5dfe6dc1e39f86c9ed1a20727fe1e5ff1d1642a4be85b0 not found: ID does not exist" Nov 01 06:02:27.560562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.560527 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 06:02:27.581528 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.581499 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d96a66-64c48bbc79-shfkd"] Nov 01 06:02:27.689951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.687966 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 06:02:27.691233 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:27.691206 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d96a66-6457d7f547-65qrf"] Nov 01 06:02:28.349285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:28.346632 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:28.349285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:28.346696 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:28.974387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:28.974358 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" path="/var/lib/kubelet/pods/55c746c6-d0ec-4333-8bdd-ccb57adfd329/volumes" Nov 01 06:02:28.974843 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:28.974817 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" path="/var/lib/kubelet/pods/b2de60f3-08d3-4146-adb6-608eb8cbcbc6/volumes" Nov 01 06:02:29.362281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:29.359721 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:29.362281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:29.359799 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:30.342583 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:30.342554 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-trkbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 06:02:30.342972 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:30.342619 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 06:02:31.346641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:31.346601 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:31.347119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:31.347081 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:31.347736 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:31.347719 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:02:40.078993 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:40.078956 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 06:02:40.079415 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:40.079160 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" containerID="cri-o://a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98" gracePeriod=30 Nov 01 06:02:40.439533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:40.439448 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:02:40.439533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:40.439519 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:02:45.348835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.348771 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 06:02:45.422291 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422236 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422366 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422399 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422449 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422470 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422513 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422566 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422610 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.422713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422625 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:02:45.422909 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422741 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:02:45.422909 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422821 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:02:45.422909 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422846 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:02:45.423064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.422982 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:02:45.423125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423068 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\") pod \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\" (UID: \"bc98c712-cfd1-4507-81ba-1c4b519ea1cd\") " Nov 01 06:02:45.423503 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423481 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.423503 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423504 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.423665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423517 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.423665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423529 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.423665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.423541 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.424872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.424851 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:02:45.424955 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.424931 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:02:45.425432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.425412 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "kube-api-access-bkjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:45.425525 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.425510 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token" pod "bc98c712-cfd1-4507-81ba-1c4b519ea1cd" (UID: "bc98c712-cfd1-4507-81ba-1c4b519ea1cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:02:45.525803 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.525424 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.525803 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.525460 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.525803 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.525476 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjfh\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-kube-api-access-bkjfh\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.525803 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.525492 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc98c712-cfd1-4507-81ba-1c4b519ea1cd-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:02:45.539292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.539230 4425 generic.go:356] "Generic (PLEG): container finished" podID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerID="a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98" exitCode=0 Nov 01 06:02:45.539456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.539294 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" Nov 01 06:02:45.539456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.539320 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerDied","Data":"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98"} Nov 01 06:02:45.539456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.539382 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn" event={"ID":"bc98c712-cfd1-4507-81ba-1c4b519ea1cd","Type":"ContainerDied","Data":"061099da50afaf47ce1488a489beb450c2d659c4c425edbb3295e9ce19b90861"} Nov 01 06:02:45.539456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.539412 4425 scope.go:122] "RemoveContainer" containerID="a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98" Nov 01 06:02:45.550187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.550157 4425 scope.go:122] "RemoveContainer" containerID="ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4" Nov 01 06:02:45.560399 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.560377 4425 scope.go:122] "RemoveContainer" containerID="a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98" Nov 01 06:02:45.560727 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:45.560701 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98\": container with ID starting with a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98 not found: ID does not exist" containerID="a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98" Nov 01 06:02:45.560816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.560740 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98"} err="failed to get container status \"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98\": rpc error: code = NotFound desc = could not find container \"a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98\": container with ID starting with a40cab4b765fe423d1a4d8044dbfeb304fc592812d2af62f8c7f3f3e829a6a98 not found: ID does not exist" Nov 01 06:02:45.560816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.560766 4425 scope.go:122] "RemoveContainer" containerID="ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4" Nov 01 06:02:45.561026 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:45.561009 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4\": container with ID starting with ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4 not found: ID does not exist" containerID="ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4" Nov 01 06:02:45.561135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.561045 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4"} err="failed to get container status \"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4\": rpc error: code = NotFound desc = could not find container \"ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4\": container with ID starting with ee9cd536d7614240c203dc333f2cb9beda8c2136bd09eb1a1da77552da9867b4 not found: ID does not exist" Nov 01 06:02:45.708515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.708424 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 06:02:45.718794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:45.718748 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d96a66-openshift-default-545c685db5-c5gdn"] Nov 01 06:02:46.974232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:46.974184 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" path="/var/lib/kubelet/pods/bc98c712-cfd1-4507-81ba-1c4b519ea1cd/volumes" Nov 01 06:02:47.599651 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:47.599616 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 06:02:47.710722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:47.710682 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 06:02:50.350307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350274 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:02:50.350838 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350823 4425 cpu_manager.go:414] "containerMap: removing container" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" Nov 01 06:02:50.350888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350839 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" Nov 01 06:02:50.350888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350852 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" Nov 01 06:02:50.350888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350858 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" Nov 01 06:02:50.350888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350872 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:02:50.350888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350877 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350896 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350901 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350920 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.350925 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.351009 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfabf84-d74e-41c4-bc9a-0c09e88c512b" containerName="payload-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.351019 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.351027 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0047c5-ad1b-4cbc-ba30-3d0d8b8b01f1" containerName="payload-pre-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.351036 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c746c6-d0ec-4333-8bdd-ccb57adfd329" containerName="payload-pre-processing" Nov 01 06:02:50.351050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.351043 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2de60f3-08d3-4146-adb6-608eb8cbcbc6" containerName="payload-processing" Nov 01 06:02:50.355930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.355909 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.412333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.412297 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:02:50.459149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.459110 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-fd2672\"/\"openshift-service-ca.crt\"" Nov 01 06:02:50.459383 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.459321 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-fd2672\"/\"kube-root-ca.crt\"" Nov 01 06:02:50.460130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.460113 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-fd2672\"/\"sub-test-model-fd2672-kserve-self-signed-certs\"" Nov 01 06:02:50.460394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.460376 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-fd2672\"/\"default-dockercfg-svnsw\"" Nov 01 06:02:50.485851 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.485814 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.486043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.485941 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.486043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.485984 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.486171 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.486068 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.486171 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.486139 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfh6\" (UniqueName: \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.486431 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.486192 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587103 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587070 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587117 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587201 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587303 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfh6\" (UniqueName: \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587353 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587422 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587588 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587678 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587638 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.587786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.587764 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.589815 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.589793 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.590201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.590177 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.649679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.649602 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfh6\" (UniqueName: \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\") pod \"sub-test-model-fd2672-kserve-7bf65989d7-6nlsc\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:50.668746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:50.668717 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:51.049577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:51.049549 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:02:51.049982 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:02:51.049955 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70255189_5f34_4c75_9689_b523fecb5bef.slice/crio-95e4673119ec629e157fcc212ad9319507307097b0127a27efe724589dc58236 WatchSource:0}: Error finding container 95e4673119ec629e157fcc212ad9319507307097b0127a27efe724589dc58236: Status 404 returned error can't find the container with id 95e4673119ec629e157fcc212ad9319507307097b0127a27efe724589dc58236 Nov 01 06:02:51.586021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:51.585981 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" event={"ID":"70255189-5f34-4c75-9689-b523fecb5bef","Type":"ContainerStarted","Data":"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff"} Nov 01 06:02:51.586456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:51.586035 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" event={"ID":"70255189-5f34-4c75-9689-b523fecb5bef","Type":"ContainerStarted","Data":"95e4673119ec629e157fcc212ad9319507307097b0127a27efe724589dc58236"} Nov 01 06:02:51.669311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:51.669277 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:02:51.706966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:51.706897 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" podStartSLOduration=1.706877089 podStartE2EDuration="1.706877089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:51.703831937 +0000 UTC m=+3586.844420858" watchObservedRunningTime="2026-11-01 06:02:51.706877089 +0000 UTC m=+3586.847466010" Nov 01 06:02:52.609465 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:52.609416 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" containerID="cri-o://67b9be07648e94e68844ae32c8eb540ce171cb7b42e89c80794f394129057730" gracePeriod=25 Nov 01 06:02:52.711042 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:52.710998 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" containerID="cri-o://eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12" gracePeriod=25 Nov 01 06:02:55.046949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:55.046914 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd"] Nov 01 06:02:55.097622 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:55.097588 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a61048-ldrdd"] Nov 01 06:02:56.101100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:56.101052 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:56.101535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:56.101122 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:02:56.947602 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:02:56.947567 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:02:56.973307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:56.973272 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3162c323-1a1a-4c3b-afad-8f00415f75cb" path="/var/lib/kubelet/pods/3162c323-1a1a-4c3b-afad-8f00415f75cb/volumes" Nov 01 06:02:59.617772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:59.617742 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 06:02:59.618285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:59.617913 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" containerID="cri-o://0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa" gracePeriod=30 Nov 01 06:02:59.654914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:59.654877 4425 patch_prober.go:29] interesting pod/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:02:59.655091 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:02:59.654938 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:03:00.669138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:00.669105 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:03:00.681892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:00.681869 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:03:01.680873 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:01.680837 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:03:04.881923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.881896 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 06:03:04.960130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960046 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960150 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960188 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960205 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960225 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960329 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960394 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960389 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960443 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960470 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\") pod \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\" (UID: \"767dedf1-1554-4fb3-a527-f4c432e5fc3d\") " Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960480 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960632 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:04.960706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960653 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:04.961298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960975 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:04.961298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.960995 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.961298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.961082 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.961298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.961098 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.961298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.961153 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:04.962776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.962748 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:03:04.962776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.962762 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "kube-api-access-nlvvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:04.962914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.962828 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:04.962914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:04.962838 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token" pod "767dedf1-1554-4fb3-a527-f4c432e5fc3d" (UID: "767dedf1-1554-4fb3-a527-f4c432e5fc3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:05.062635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.062596 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.062635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.062634 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.062901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.062650 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.062901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.062665 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvvp\" (UniqueName: \"kubernetes.io/projected/767dedf1-1554-4fb3-a527-f4c432e5fc3d-kube-api-access-nlvvp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.062901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.062680 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767dedf1-1554-4fb3-a527-f4c432e5fc3d-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:05.703168 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.703131 4425 generic.go:356] "Generic (PLEG): container finished" podID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerID="0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa" exitCode=0 Nov 01 06:03:05.703388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.703194 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" Nov 01 06:03:05.703388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.703223 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerDied","Data":"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa"} Nov 01 06:03:05.703388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.703318 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7" event={"ID":"767dedf1-1554-4fb3-a527-f4c432e5fc3d","Type":"ContainerDied","Data":"fe03aeabd16097124487c3dba6c3dea544d09c62ebcc017cde0d513df08bd60a"} Nov 01 06:03:05.703388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.703347 4425 scope.go:122] "RemoveContainer" containerID="0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa" Nov 01 06:03:05.713551 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.713533 4425 scope.go:122] "RemoveContainer" containerID="84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a" Nov 01 06:03:05.723186 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.723159 4425 scope.go:122] "RemoveContainer" containerID="0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa" Nov 01 06:03:05.723499 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:05.723480 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa\": container with ID starting with 0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa not found: ID does not exist" containerID="0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa" Nov 01 06:03:05.723575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.723517 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa"} err="failed to get container status \"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa\": rpc error: code = NotFound desc = could not find container \"0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa\": container with ID starting with 0b52bbe70eca7134d6623233a5d9f40dcb135b85935d9f549098662b224376fa not found: ID does not exist" Nov 01 06:03:05.723575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.723536 4425 scope.go:122] "RemoveContainer" containerID="84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a" Nov 01 06:03:05.723827 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:05.723807 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a\": container with ID starting with 84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a not found: ID does not exist" containerID="84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a" Nov 01 06:03:05.723881 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.723835 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a"} err="failed to get container status \"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a\": rpc error: code = NotFound desc = could not find container \"84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a\": container with ID starting with 84b1cb16e076ee3147fd97a614db9ff8857a4dad5b637d14f04c84cc5eaefb9a not found: ID does not exist" Nov 01 06:03:05.922831 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.922799 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 06:03:05.926218 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:05.926191 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-dee6aa-openshift-default-7785b59785-gt2w7"] Nov 01 06:03:06.047493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.047397 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4"] Nov 01 06:03:06.050473 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.050449 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6749c390-4vct4"] Nov 01 06:03:06.101019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.100982 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:06.101206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.101080 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:06.974011 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.973978 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" path="/var/lib/kubelet/pods/767dedf1-1554-4fb3-a527-f4c432e5fc3d/volumes" Nov 01 06:03:06.974476 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:06.974461 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6957ee8-4aa6-44a9-8db4-1ae6b0892119" path="/var/lib/kubelet/pods/a6957ee8-4aa6-44a9-8db4-1ae6b0892119/volumes" Nov 01 06:03:12.773578 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:12.773544 4425 generic.go:356] "Generic (PLEG): container finished" podID="4901c925-1e05-4884-a9a1-e717453b340f" containerID="67b9be07648e94e68844ae32c8eb540ce171cb7b42e89c80794f394129057730" exitCode=0 Nov 01 06:03:12.773971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:12.773619 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" event={"ID":"4901c925-1e05-4884-a9a1-e717453b340f","Type":"ContainerDied","Data":"67b9be07648e94e68844ae32c8eb540ce171cb7b42e89c80794f394129057730"} Nov 01 06:03:12.978498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:12.978472 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 06:03:13.028418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.028388 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 06:03:13.073198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.073157 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\" (UniqueName: \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\") pod \"4901c925-1e05-4884-a9a1-e717453b340f\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " Nov 01 06:03:13.073437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.073229 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\") pod \"4901c925-1e05-4884-a9a1-e717453b340f\" (UID: \"4901c925-1e05-4884-a9a1-e717453b340f\") " Nov 01 06:03:13.073657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.073628 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume" pod "4901c925-1e05-4884-a9a1-e717453b340f" (UID: "4901c925-1e05-4884-a9a1-e717453b340f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:13.073958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.073932 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4901c925-1e05-4884-a9a1-e717453b340f-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:13.075641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.075615 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t" pod "4901c925-1e05-4884-a9a1-e717453b340f" (UID: "4901c925-1e05-4884-a9a1-e717453b340f"). InnerVolumeSpecName "kube-api-access-n5x6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:13.174716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.174629 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\") pod \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " Nov 01 06:03:13.174716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.174686 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\" (UniqueName: \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\") pod \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\" (UID: \"9eb19815-0b0d-47d7-8721-dbd15eafcf93\") " Nov 01 06:03:13.175017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.174994 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume" pod "9eb19815-0b0d-47d7-8721-dbd15eafcf93" (UID: "9eb19815-0b0d-47d7-8721-dbd15eafcf93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:13.175068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.175018 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5x6t\" (UniqueName: \"kubernetes.io/projected/4901c925-1e05-4884-a9a1-e717453b340f-kube-api-access-n5x6t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:13.177017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.176995 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb" pod "9eb19815-0b0d-47d7-8721-dbd15eafcf93" (UID: "9eb19815-0b0d-47d7-8721-dbd15eafcf93"). InnerVolumeSpecName "kube-api-access-64xhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:13.275779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.275742 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb19815-0b0d-47d7-8721-dbd15eafcf93-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:13.275779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.275775 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xhb\" (UniqueName: \"kubernetes.io/projected/9eb19815-0b0d-47d7-8721-dbd15eafcf93-kube-api-access-64xhb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:13.781225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.781188 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" Nov 01 06:03:13.781225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.781200 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt" event={"ID":"4901c925-1e05-4884-a9a1-e717453b340f","Type":"ContainerDied","Data":"6ccdb30e5098b0d792cf46290165e1a9462e19c40d22cfd6268466682a219dec"} Nov 01 06:03:13.781789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.781325 4425 scope.go:122] "RemoveContainer" containerID="67b9be07648e94e68844ae32c8eb540ce171cb7b42e89c80794f394129057730" Nov 01 06:03:13.782826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.782797 4425 generic.go:356] "Generic (PLEG): container finished" podID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerID="eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12" exitCode=0 Nov 01 06:03:13.783144 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.782836 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" Nov 01 06:03:13.783144 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.782866 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" event={"ID":"9eb19815-0b0d-47d7-8721-dbd15eafcf93","Type":"ContainerDied","Data":"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12"} Nov 01 06:03:13.783144 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.782910 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh" event={"ID":"9eb19815-0b0d-47d7-8721-dbd15eafcf93","Type":"ContainerDied","Data":"bc6a41f3f24919d3d8816d55f68750a3edd974f71df7a3cb614f8413da3263a4"} Nov 01 06:03:13.791487 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.791466 4425 scope.go:122] "RemoveContainer" containerID="eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12" Nov 01 06:03:13.801220 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.801200 4425 scope.go:122] "RemoveContainer" containerID="eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12" Nov 01 06:03:13.801568 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:13.801549 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12\": container with ID starting with eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12 not found: ID does not exist" containerID="eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12" Nov 01 06:03:13.801625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.801577 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12"} err="failed to get container status \"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12\": rpc error: code = NotFound desc = could not find container \"eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12\": container with ID starting with eeb959e32f78cbce9e34e706e9d6ded15fecea074eda8be426d65003cc0a5d12 not found: ID does not exist" Nov 01 06:03:13.949172 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.949027 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 06:03:13.999777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:13.999741 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-098e4d-5f8b9f6575-lngfh"] Nov 01 06:03:14.070225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:14.070186 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 06:03:14.073772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:14.073746 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-098e4d-58d9df76b9-7drbt"] Nov 01 06:03:14.974810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:14.974779 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4901c925-1e05-4884-a9a1-e717453b340f" path="/var/lib/kubelet/pods/4901c925-1e05-4884-a9a1-e717453b340f/volumes" Nov 01 06:03:14.975222 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:14.975180 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" path="/var/lib/kubelet/pods/9eb19815-0b0d-47d7-8721-dbd15eafcf93/volumes" Nov 01 06:03:15.764274 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764223 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw"] Nov 01 06:03:15.764851 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764834 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" Nov 01 06:03:15.764910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764853 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" Nov 01 06:03:15.764910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764873 4425 cpu_manager.go:414] "containerMap: removing container" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:15.764910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764878 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:15.764910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764896 4425 cpu_manager.go:414] "containerMap: removing container" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:15.764910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764902 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764917 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764922 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764933 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.764938 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.765023 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4901c925-1e05-4884-a9a1-e717453b340f" containerName="payload-processing" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.765032 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc98c712-cfd1-4507-81ba-1c4b519ea1cd" containerName="istio-proxy" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.765042 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:15.765109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.765052 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb19815-0b0d-47d7-8721-dbd15eafcf93" containerName="payload-pre-processing" Nov 01 06:03:15.769941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.769920 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:15.825771 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.825737 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw"] Nov 01 06:03:15.872625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.872587 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:15.872625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.872590 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:15.872852 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.872632 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:15.873842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.873824 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:03:15.910237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.910205 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr7d\" (UniqueName: \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:15.910441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.910408 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:15.910441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:15.910430 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.011634 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.011595 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr7d\" (UniqueName: \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.012052 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.011728 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.012052 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.011749 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.012515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.012487 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.014429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.014369 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.073715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.073673 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr7d\" (UniqueName: \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\") pod \"maas-api-revoke-sub-6378ec37-lbdgw\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.084360 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.084334 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:16.462392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.462364 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw"] Nov 01 06:03:16.462692 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:03:16.462670 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aadf8b3_0542_44f2_87e8_80c7dccf1530.slice/crio-e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667 WatchSource:0}: Error finding container e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667: Status 404 returned error can't find the container with id e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667 Nov 01 06:03:16.805681 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:16.805593 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" event={"ID":"3aadf8b3-0542-44f2-87e8-80c7dccf1530","Type":"ContainerStarted","Data":"e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667"} Nov 01 06:03:17.813922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:17.813883 4425 generic.go:356] "Generic (PLEG): container finished" podID="3aadf8b3-0542-44f2-87e8-80c7dccf1530" containerID="acd5cbe3de6d8c2d4ed60d35592daaba2634209ca4f28d923ac536f4f132495e" exitCode=0 Nov 01 06:03:17.814430 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:17.813975 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" event={"ID":"3aadf8b3-0542-44f2-87e8-80c7dccf1530","Type":"ContainerDied","Data":"acd5cbe3de6d8c2d4ed60d35592daaba2634209ca4f28d923ac536f4f132495e"} Nov 01 06:03:18.981010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:18.980984 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:19.150112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.150035 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\" (UniqueName: \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\") pod \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " Nov 01 06:03:19.150112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.150106 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\") pod \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " Nov 01 06:03:19.150338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.150187 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\") pod \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\" (UID: \"3aadf8b3-0542-44f2-87e8-80c7dccf1530\") " Nov 01 06:03:19.150594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.150561 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca" pod "3aadf8b3-0542-44f2-87e8-80c7dccf1530" (UID: "3aadf8b3-0542-44f2-87e8-80c7dccf1530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:19.150890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.150867 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:19.152527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.152487 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca" pod "3aadf8b3-0542-44f2-87e8-80c7dccf1530" (UID: "3aadf8b3-0542-44f2-87e8-80c7dccf1530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:19.152638 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.152563 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d" pod "3aadf8b3-0542-44f2-87e8-80c7dccf1530" (UID: "3aadf8b3-0542-44f2-87e8-80c7dccf1530"). InnerVolumeSpecName "kube-api-access-wrr7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:19.252665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.252619 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr7d\" (UniqueName: \"kubernetes.io/projected/3aadf8b3-0542-44f2-87e8-80c7dccf1530-kube-api-access-wrr7d\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:19.252665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.252663 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aadf8b3-0542-44f2-87e8-80c7dccf1530-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:19.828654 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.828623 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" Nov 01 06:03:19.828826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.828658 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw" event={"ID":"3aadf8b3-0542-44f2-87e8-80c7dccf1530","Type":"ContainerDied","Data":"e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667"} Nov 01 06:03:19.828826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:19.828704 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a2555e51daa28c61035b6ca4aa18422555c077742cfa61630d1e153552c667" Nov 01 06:03:20.139484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:20.139398 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 06:03:21.263491 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:21.263453 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 06:03:21.263972 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:21.263726 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" containerID="cri-o://7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac" gracePeriod=30 Nov 01 06:03:21.732871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:21.732838 4425 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:03:21.733049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:21.732895 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:03:24.049384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:24.049350 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp"] Nov 01 06:03:24.101266 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:24.101218 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1607c6b-d6khp"] Nov 01 06:03:24.973966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:24.973935 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208a62c2-2057-4e6a-9810-b9562441e759" path="/var/lib/kubelet/pods/208a62c2-2057-4e6a-9810-b9562441e759/volumes" Nov 01 06:03:26.531724 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.531653 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 06:03:26.639880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.639848 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.639954 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.639974 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.639992 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640025 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640073 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640099 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640181 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640208 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\") pod \"0be31ef2-1642-4b6b-b69e-c3532095a677\" (UID: \"0be31ef2-1642-4b6b-b69e-c3532095a677\") " Nov 01 06:03:26.640349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640307 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:26.640620 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640554 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:26.640620 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640606 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:26.640740 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640657 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:26.640965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640824 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.640965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640846 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:26.641100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640968 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.641100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.640986 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.641100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.641001 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be31ef2-1642-4b6b-b69e-c3532095a677-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.642546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.642524 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:03:26.642677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.642569 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:26.642677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.642605 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:26.642758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.642713 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv" pod "0be31ef2-1642-4b6b-b69e-c3532095a677" (UID: "0be31ef2-1642-4b6b-b69e-c3532095a677"). InnerVolumeSpecName "kube-api-access-7tvnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:26.742266 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.742210 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.742266 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.742266 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvnv\" (UniqueName: \"kubernetes.io/projected/0be31ef2-1642-4b6b-b69e-c3532095a677-kube-api-access-7tvnv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.742506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.742284 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.742506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.742298 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.742506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.742311 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be31ef2-1642-4b6b-b69e-c3532095a677-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:26.871460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.871424 4425 generic.go:356] "Generic (PLEG): container finished" podID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerID="7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac" exitCode=0 Nov 01 06:03:26.871682 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.871476 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" Nov 01 06:03:26.871682 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.871511 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" event={"ID":"0be31ef2-1642-4b6b-b69e-c3532095a677","Type":"ContainerDied","Data":"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac"} Nov 01 06:03:26.871682 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.871582 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p" event={"ID":"0be31ef2-1642-4b6b-b69e-c3532095a677","Type":"ContainerDied","Data":"236396752b13960da4a469b7515e282a3039f49d1a0111e84ffb4540f5cfb2b8"} Nov 01 06:03:26.871682 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.871615 4425 scope.go:122] "RemoveContainer" containerID="7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac" Nov 01 06:03:26.883115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.883092 4425 scope.go:122] "RemoveContainer" containerID="7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac" Nov 01 06:03:26.883438 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:26.883420 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac\": container with ID starting with 7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac not found: ID does not exist" containerID="7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac" Nov 01 06:03:26.883502 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:26.883446 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac"} err="failed to get container status \"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac\": rpc error: code = NotFound desc = could not find container \"7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac\": container with ID starting with 7203050cbda4c1900ec5418eef2995f200fa84174cef2a0a10920c2f4a8556ac not found: ID does not exist" Nov 01 06:03:26.949059 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:26.949021 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:03:27.043601 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:27.043565 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 06:03:27.047101 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:27.047077 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-xww2p"] Nov 01 06:03:28.974348 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:28.974308 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" path="/var/lib/kubelet/pods/0be31ef2-1642-4b6b-b69e-c3532095a677/volumes" Nov 01 06:03:31.919390 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.919353 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt"] Nov 01 06:03:31.920128 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920105 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" Nov 01 06:03:31.920241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920131 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" Nov 01 06:03:31.920241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920153 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3aadf8b3-0542-44f2-87e8-80c7dccf1530" containerName="revoke-keys" Nov 01 06:03:31.920241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920162 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aadf8b3-0542-44f2-87e8-80c7dccf1530" containerName="revoke-keys" Nov 01 06:03:31.920471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920322 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aadf8b3-0542-44f2-87e8-80c7dccf1530" containerName="revoke-keys" Nov 01 06:03:31.920471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920342 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="767dedf1-1554-4fb3-a527-f4c432e5fc3d" containerName="istio-proxy" Nov 01 06:03:31.920471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.920365 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be31ef2-1642-4b6b-b69e-c3532095a677" containerName="istio-proxy" Nov 01 06:03:31.925232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.925211 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:31.980927 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:31.980893 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt"] Nov 01 06:03:32.003426 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.003385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.003642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.003450 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wtx\" (UniqueName: \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.003642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.003562 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.027760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.027704 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:03:32.027760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.027735 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:03:32.028012 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.027762 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:03:32.029315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.029293 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:03:32.104811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.104770 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.105038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.104828 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wtx\" (UniqueName: \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.105038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.104935 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.105607 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.105579 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.107586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.107564 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.167910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.167874 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wtx\" (UniqueName: \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\") pod \"maas-api-revoke-sub-ff5b6f0d-ftkbt\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.239488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.239384 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:32.350200 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.350159 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 06:03:32.458325 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.458289 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 06:03:32.619548 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.619515 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt"] Nov 01 06:03:32.620082 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:03:32.620058 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95198ef_c5d3_42a1_8716_92fbfe2865fb.slice/crio-3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15 WatchSource:0}: Error finding container 3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15: Status 404 returned error can't find the container with id 3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15 Nov 01 06:03:32.922901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:32.922807 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" event={"ID":"f95198ef-c5d3-42a1-8716-92fbfe2865fb","Type":"ContainerStarted","Data":"3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15"} Nov 01 06:03:33.931602 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:33.931554 4425 generic.go:356] "Generic (PLEG): container finished" podID="f95198ef-c5d3-42a1-8716-92fbfe2865fb" containerID="08aaf1a810881281ef9941588766319b32431946b2f7aa9d6fb9ec6dcf8666ab" exitCode=0 Nov 01 06:03:33.932044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:33.931642 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" event={"ID":"f95198ef-c5d3-42a1-8716-92fbfe2865fb","Type":"ContainerDied","Data":"08aaf1a810881281ef9941588766319b32431946b2f7aa9d6fb9ec6dcf8666ab"} Nov 01 06:03:35.096390 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.096356 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:35.139363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.139326 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\" (UniqueName: \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\") pod \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " Nov 01 06:03:35.139533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.139399 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\") pod \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " Nov 01 06:03:35.139533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.139526 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\") pod \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\" (UID: \"f95198ef-c5d3-42a1-8716-92fbfe2865fb\") " Nov 01 06:03:35.139920 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.139894 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca" pod "f95198ef-c5d3-42a1-8716-92fbfe2865fb" (UID: "f95198ef-c5d3-42a1-8716-92fbfe2865fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:35.140182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.139968 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.141794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.141762 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx" pod "f95198ef-c5d3-42a1-8716-92fbfe2865fb" (UID: "f95198ef-c5d3-42a1-8716-92fbfe2865fb"). InnerVolumeSpecName "kube-api-access-c2wtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:35.141902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.141819 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca" pod "f95198ef-c5d3-42a1-8716-92fbfe2865fb" (UID: "f95198ef-c5d3-42a1-8716-92fbfe2865fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:35.161050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.161018 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" containerID="cri-o://3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee" gracePeriod=45 Nov 01 06:03:35.240830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.240745 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wtx\" (UniqueName: \"kubernetes.io/projected/f95198ef-c5d3-42a1-8716-92fbfe2865fb-kube-api-access-c2wtx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.240830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.240791 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95198ef-c5d3-42a1-8716-92fbfe2865fb-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.404721 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.404693 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 06:03:35.543788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.543694 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.543788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.543758 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.544046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.543836 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.544046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.543869 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\" (UniqueName: \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.544046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.543953 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.544046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544001 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:35.544292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544058 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\" (UniqueName: \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\") pod \"40481906-1eb4-484a-81f7-65ab5940aaca\" (UID: \"40481906-1eb4-484a-81f7-65ab5940aaca\") " Nov 01 06:03:35.544292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544075 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:35.544292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544086 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:35.544731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544657 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.544731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544684 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.544731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.544699 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.546931 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.546901 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:03:35.547036 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.546937 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:35.547036 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.546938 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr" pod "40481906-1eb4-484a-81f7-65ab5940aaca" (UID: "40481906-1eb4-484a-81f7-65ab5940aaca"). InnerVolumeSpecName "kube-api-access-gdkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:35.645385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.645353 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdkbr\" (UniqueName: \"kubernetes.io/projected/40481906-1eb4-484a-81f7-65ab5940aaca-kube-api-access-gdkbr\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.645385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.645384 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40481906-1eb4-484a-81f7-65ab5940aaca-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.645606 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.645394 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40481906-1eb4-484a-81f7-65ab5940aaca-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:35.946539 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.946510 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" Nov 01 06:03:35.946760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.946557 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt" event={"ID":"f95198ef-c5d3-42a1-8716-92fbfe2865fb","Type":"ContainerDied","Data":"3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15"} Nov 01 06:03:35.946760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.946618 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2be553421541e16dd7a0481992b618822ad41e1311604aedddae066f4b8c15" Nov 01 06:03:35.948146 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.948114 4425 generic.go:356] "Generic (PLEG): container finished" podID="40481906-1eb4-484a-81f7-65ab5940aaca" containerID="3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee" exitCode=0 Nov 01 06:03:35.948307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.948165 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" Nov 01 06:03:35.948307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.948210 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" event={"ID":"40481906-1eb4-484a-81f7-65ab5940aaca","Type":"ContainerDied","Data":"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee"} Nov 01 06:03:35.948307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.948271 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8" event={"ID":"40481906-1eb4-484a-81f7-65ab5940aaca","Type":"ContainerDied","Data":"11c0689ceec61d94209ed69a4178f7dc0d2dc1fdea5acca735bb3c6cc1043f84"} Nov 01 06:03:35.948307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.948297 4425 scope.go:122] "RemoveContainer" containerID="3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee" Nov 01 06:03:35.959732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.959557 4425 scope.go:122] "RemoveContainer" containerID="3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee" Nov 01 06:03:35.959995 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:35.959968 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee\": container with ID starting with 3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee not found: ID does not exist" containerID="3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee" Nov 01 06:03:35.960066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:35.960005 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee"} err="failed to get container status \"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee\": rpc error: code = NotFound desc = could not find container \"3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee\": container with ID starting with 3b560774f9fab8ee302ee93ee8c6baac4650530fc085628400d6483552e626ee not found: ID does not exist" Nov 01 06:03:36.052464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.052426 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2"] Nov 01 06:03:36.055577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.055544 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2071016-gf2m2"] Nov 01 06:03:36.116386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.116348 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 06:03:36.120204 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.120173 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/distinct-w4-a48099-kserve-55547bd958-dqcg8"] Nov 01 06:03:36.973789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.973754 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e706358-9563-482d-837d-0d067ded516a" path="/var/lib/kubelet/pods/0e706358-9563-482d-837d-0d067ded516a/volumes" Nov 01 06:03:36.974212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:36.974197 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" path="/var/lib/kubelet/pods/40481906-1eb4-484a-81f7-65ab5940aaca/volumes" Nov 01 06:03:37.350731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:37.350682 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" containerID="cri-o://de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465" gracePeriod=25 Nov 01 06:03:37.459302 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:37.459246 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" containerID="cri-o://7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6" gracePeriod=25 Nov 01 06:03:41.158352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:41.158313 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:41.158352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:41.158329 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:47.506112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:47.506019 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 06:03:47.506741 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:47.506299 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" containerID="cri-o://db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f" gracePeriod=30 Nov 01 06:03:48.048524 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:48.048497 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr"] Nov 01 06:03:48.051388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:48.051367 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cb0565-whvwr"] Nov 01 06:03:48.973866 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:48.973831 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0300dffd-9474-48e3-9e97-46cde7d46c99" path="/var/lib/kubelet/pods/0300dffd-9474-48e3-9e97-46cde7d46c99/volumes" Nov 01 06:03:51.158840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:51.158791 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:51.158840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:51.158831 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:03:52.812952 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.812921 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 06:03:52.924337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924306 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924353 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924398 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924464 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924529 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924569 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.924696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924655 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924727 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924818 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\") pod \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\" (UID: \"7344589a-0c6e-4ed8-8075-8e88dd1b51d4\") " Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924814 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924858 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.924992 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925053 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:52.925328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925219 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:52.925953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925536 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:52.925953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925560 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:52.925953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925576 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:52.925953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925592 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:52.925953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.925607 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:52.926898 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.926874 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:03:52.927352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.927327 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:52.927478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.927403 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "kube-api-access-7f92g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:52.927478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:52.927441 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy" pod "7344589a-0c6e-4ed8-8075-8e88dd1b51d4" (UID: "7344589a-0c6e-4ed8-8075-8e88dd1b51d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:03:53.026765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.026727 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:53.026765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.026763 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:53.027020 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.026779 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:53.027020 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.026793 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f92g\" (UniqueName: \"kubernetes.io/projected/7344589a-0c6e-4ed8-8075-8e88dd1b51d4-kube-api-access-7f92g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:53.071494 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.071460 4425 generic.go:356] "Generic (PLEG): container finished" podID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerID="db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f" exitCode=0 Nov 01 06:03:53.071679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.071520 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" Nov 01 06:03:53.071679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.071545 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerDied","Data":"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f"} Nov 01 06:03:53.071679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.071609 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2" event={"ID":"7344589a-0c6e-4ed8-8075-8e88dd1b51d4","Type":"ContainerDied","Data":"05c9ee58c15a0c79515f9da43afdc1e27b70e54f886f50758c6a776ef1083031"} Nov 01 06:03:53.071679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.071632 4425 scope.go:122] "RemoveContainer" containerID="db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f" Nov 01 06:03:53.081743 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.081725 4425 scope.go:122] "RemoveContainer" containerID="31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea" Nov 01 06:03:53.091450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.091430 4425 scope.go:122] "RemoveContainer" containerID="db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f" Nov 01 06:03:53.091766 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:53.091745 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f\": container with ID starting with db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f not found: ID does not exist" containerID="db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f" Nov 01 06:03:53.091863 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.091780 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f"} err="failed to get container status \"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f\": rpc error: code = NotFound desc = could not find container \"db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f\": container with ID starting with db6242f89f26f6d28c6216655089dd576f28811f57c4be14f3aeae08376d831f not found: ID does not exist" Nov 01 06:03:53.091863 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.091808 4425 scope.go:122] "RemoveContainer" containerID="31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea" Nov 01 06:03:53.092085 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:53.092066 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea\": container with ID starting with 31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea not found: ID does not exist" containerID="31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea" Nov 01 06:03:53.092130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.092094 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea"} err="failed to get container status \"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea\": rpc error: code = NotFound desc = could not find container \"31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea\": container with ID starting with 31e534f3d362430e558930db4c713fc06682854efe3c7dc18bb84e0ba8273cea not found: ID does not exist" Nov 01 06:03:53.240582 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.240552 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 06:03:53.246593 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:53.246563 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-098e4d-openshift-default-dbccdd77c-ptlf2"] Nov 01 06:03:54.973411 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:54.973379 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" path="/var/lib/kubelet/pods/7344589a-0c6e-4ed8-8075-8e88dd1b51d4/volumes" Nov 01 06:03:56.948399 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:56.948351 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:03:57.624671 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.624605 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 06:03:57.684437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.684403 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\") pod \"907e94f8-1acf-4653-9d52-d6af35d9097c\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " Nov 01 06:03:57.684648 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.684546 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\" (UniqueName: \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\") pod \"907e94f8-1acf-4653-9d52-d6af35d9097c\" (UID: \"907e94f8-1acf-4653-9d52-d6af35d9097c\") " Nov 01 06:03:57.685568 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.685535 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume" pod "907e94f8-1acf-4653-9d52-d6af35d9097c" (UID: "907e94f8-1acf-4653-9d52-d6af35d9097c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:57.687179 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.687150 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d" pod "907e94f8-1acf-4653-9d52-d6af35d9097c" (UID: "907e94f8-1acf-4653-9d52-d6af35d9097c"). InnerVolumeSpecName "kube-api-access-shp2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:57.710778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.710749 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 06:03:57.785853 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.785825 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\" (UniqueName: \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\") pod \"d06eda37-d96c-4770-a9a2-80ad231053a9\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " Nov 01 06:03:57.786043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.785962 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\") pod \"d06eda37-d96c-4770-a9a2-80ad231053a9\" (UID: \"d06eda37-d96c-4770-a9a2-80ad231053a9\") " Nov 01 06:03:57.786444 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.786420 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume" pod "d06eda37-d96c-4770-a9a2-80ad231053a9" (UID: "d06eda37-d96c-4770-a9a2-80ad231053a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:03:57.786546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.786449 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shp2d\" (UniqueName: \"kubernetes.io/projected/907e94f8-1acf-4653-9d52-d6af35d9097c-kube-api-access-shp2d\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:57.786546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.786466 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907e94f8-1acf-4653-9d52-d6af35d9097c-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:57.788159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.788134 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8" pod "d06eda37-d96c-4770-a9a2-80ad231053a9" (UID: "d06eda37-d96c-4770-a9a2-80ad231053a9"). InnerVolumeSpecName "kube-api-access-ttxz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:03:57.887513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.887437 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06eda37-d96c-4770-a9a2-80ad231053a9-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:57.887513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:57.887467 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxz8\" (UniqueName: \"kubernetes.io/projected/d06eda37-d96c-4770-a9a2-80ad231053a9-kube-api-access-ttxz8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:03:58.104435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.104401 4425 generic.go:356] "Generic (PLEG): container finished" podID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerID="de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465" exitCode=0 Nov 01 06:03:58.104895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.104450 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" Nov 01 06:03:58.104895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.104491 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" event={"ID":"907e94f8-1acf-4653-9d52-d6af35d9097c","Type":"ContainerDied","Data":"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465"} Nov 01 06:03:58.104895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.104544 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx" event={"ID":"907e94f8-1acf-4653-9d52-d6af35d9097c","Type":"ContainerDied","Data":"ffc0af56e99b05a1e90feef14459fe5992f44f5ecc236a8863523b405747c2eb"} Nov 01 06:03:58.104895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.104566 4425 scope.go:122] "RemoveContainer" containerID="de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465" Nov 01 06:03:58.106152 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.106129 4425 generic.go:356] "Generic (PLEG): container finished" podID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerID="7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6" exitCode=0 Nov 01 06:03:58.106330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.106193 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" Nov 01 06:03:58.106330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.106215 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" event={"ID":"d06eda37-d96c-4770-a9a2-80ad231053a9","Type":"ContainerDied","Data":"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6"} Nov 01 06:03:58.106330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.106304 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp" event={"ID":"d06eda37-d96c-4770-a9a2-80ad231053a9","Type":"ContainerDied","Data":"dcca352d4ff1b9280d59beaa7c96e6147b16f25efe14d264f2fa8bae1ee7c99b"} Nov 01 06:03:58.115150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.115045 4425 scope.go:122] "RemoveContainer" containerID="de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465" Nov 01 06:03:58.115591 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:58.115567 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465\": container with ID starting with de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465 not found: ID does not exist" containerID="de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465" Nov 01 06:03:58.115718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.115598 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465"} err="failed to get container status \"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465\": rpc error: code = NotFound desc = could not find container \"de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465\": container with ID starting with de04447f3dc64096d3b6a53ac0fd49818b133712701f8d55b9330e60da4db465 not found: ID does not exist" Nov 01 06:03:58.115718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.115614 4425 scope.go:122] "RemoveContainer" containerID="7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6" Nov 01 06:03:58.124363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.124343 4425 scope.go:122] "RemoveContainer" containerID="7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6" Nov 01 06:03:58.124633 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:03:58.124614 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6\": container with ID starting with 7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6 not found: ID does not exist" containerID="7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6" Nov 01 06:03:58.124703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.124640 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6"} err="failed to get container status \"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6\": rpc error: code = NotFound desc = could not find container \"7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6\": container with ID starting with 7b0ae169370ea9bfe55d7283e6816c5dcabe40889ca93a89b78295bf435b7cf6 not found: ID does not exist" Nov 01 06:03:58.273634 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.273554 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 06:03:58.277373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.277351 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a48099-6566d45677-7kcpx"] Nov 01 06:03:58.388416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.388379 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 06:03:58.392645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.392619 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a48099-786fbc588c-zpqsp"] Nov 01 06:03:58.972635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.972592 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" path="/var/lib/kubelet/pods/907e94f8-1acf-4653-9d52-d6af35d9097c/volumes" Nov 01 06:03:58.973126 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:03:58.973103 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" path="/var/lib/kubelet/pods/d06eda37-d96c-4770-a9a2-80ad231053a9/volumes" Nov 01 06:04:01.088269 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:01.088223 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 06:04:01.088695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:01.088457 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" containerID="cri-o://f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0" gracePeriod=30 Nov 01 06:04:03.047792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:03.047749 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf"] Nov 01 06:04:03.098370 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:03.098337 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cbf8dbb-pdsrf"] Nov 01 06:04:04.973321 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:04.973288 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64b4217-8e8e-416d-9570-d8311c936a1f" path="/var/lib/kubelet/pods/c64b4217-8e8e-416d-9570-d8311c936a1f/volumes" Nov 01 06:04:06.347908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.347882 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 06:04:06.485792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485697 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.485792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485773 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485827 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485895 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485946 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.485993 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486020 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486031 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486062 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486135 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486193 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\") pod \"764e847b-5005-4bc2-b50f-af824b3d68a6\" (UID: \"764e847b-5005-4bc2-b50f-af824b3d68a6\") " Nov 01 06:04:06.486361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486331 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486367 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486386 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486638 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486665 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486680 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/764e847b-5005-4bc2-b50f-af824b3d68a6-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.486697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486694 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.486938 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.486708 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.488394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.488373 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:06.488648 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.488622 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:06.488860 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.488835 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:06.488943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.488901 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb" pod "764e847b-5005-4bc2-b50f-af824b3d68a6" (UID: "764e847b-5005-4bc2-b50f-af824b3d68a6"). InnerVolumeSpecName "kube-api-access-v9wlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:06.588053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.588020 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.588053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.588049 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wlb\" (UniqueName: \"kubernetes.io/projected/764e847b-5005-4bc2-b50f-af824b3d68a6-kube-api-access-v9wlb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.588053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.588059 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:06.591968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:06.588067 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/764e847b-5005-4bc2-b50f-af824b3d68a6-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:07.057803 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.057757 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq"] Nov 01 06:04:07.058033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.057831 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-478aff25-cv6rq"] Nov 01 06:04:07.171100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.171063 4425 generic.go:356] "Generic (PLEG): container finished" podID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerID="f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0" exitCode=0 Nov 01 06:04:07.171343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.171133 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" Nov 01 06:04:07.171343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.171157 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerDied","Data":"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0"} Nov 01 06:04:07.171343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.171265 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk" event={"ID":"764e847b-5005-4bc2-b50f-af824b3d68a6","Type":"ContainerDied","Data":"abe1e223c5100dbe59d52d7c06fa41d7d22feece78c06975980d8cf1263423ad"} Nov 01 06:04:07.171343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.171289 4425 scope.go:122] "RemoveContainer" containerID="f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0" Nov 01 06:04:07.180920 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.180900 4425 scope.go:122] "RemoveContainer" containerID="8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca" Nov 01 06:04:07.189995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.189977 4425 scope.go:122] "RemoveContainer" containerID="f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0" Nov 01 06:04:07.190309 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:07.190285 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0\": container with ID starting with f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0 not found: ID does not exist" containerID="f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0" Nov 01 06:04:07.190391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.190317 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0"} err="failed to get container status \"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0\": rpc error: code = NotFound desc = could not find container \"f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0\": container with ID starting with f5d5c6bd4b1e4660089e61d1ebeea2ecd029188f2de767d73ebcb807fdbc11d0 not found: ID does not exist" Nov 01 06:04:07.190391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.190339 4425 scope.go:122] "RemoveContainer" containerID="8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca" Nov 01 06:04:07.190604 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:07.190586 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca\": container with ID starting with 8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca not found: ID does not exist" containerID="8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca" Nov 01 06:04:07.190647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.190613 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca"} err="failed to get container status \"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca\": rpc error: code = NotFound desc = could not find container \"8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca\": container with ID starting with 8f7fd95fc6fe2990f1912e641b12d7d549c6f50550fffc0cc2a4c9bbafc66dca not found: ID does not exist" Nov 01 06:04:07.340099 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.340062 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 06:04:07.390887 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:07.390855 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-afc555-openshift-default-9b8595cd9-zwkgk"] Nov 01 06:04:08.019234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.019199 4425 scope.go:122] "RemoveContainer" containerID="98bdcbd56debc8b499ece90546966d94030d6f186e0e67726a8e658d2eaae4c5" Nov 01 06:04:08.029058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.029029 4425 scope.go:122] "RemoveContainer" containerID="7598e28afe15f90e329aab8919daeb0b195e7ed778ecb394045dfbb168340bdf" Nov 01 06:04:08.833948 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.833917 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834411 4425 cpu_manager.go:414] "containerMap: removing container" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834423 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834433 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834438 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834448 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834454 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834473 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.834480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834479 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834490 4425 cpu_manager.go:414] "containerMap: removing container" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834495 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834502 4425 cpu_manager.go:414] "containerMap: removing container" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834506 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834519 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f95198ef-c5d3-42a1-8716-92fbfe2865fb" containerName="revoke-keys" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834523 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95198ef-c5d3-42a1-8716-92fbfe2865fb" containerName="revoke-keys" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834604 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95198ef-c5d3-42a1-8716-92fbfe2865fb" containerName="revoke-keys" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834612 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834619 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="907e94f8-1acf-4653-9d52-d6af35d9097c" containerName="payload-processing" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834625 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834631 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06eda37-d96c-4770-a9a2-80ad231053a9" containerName="payload-pre-processing" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834640 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.834886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.834646 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="40481906-1eb4-484a-81f7-65ab5940aaca" containerName="main" Nov 01 06:04:08.839876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.839856 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:08.853206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.853179 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:08.853656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.853643 4425 cpu_manager.go:414] "containerMap: removing container" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.853708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.853657 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" containerName="istio-proxy" Nov 01 06:04:08.853776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.853767 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7344589a-0c6e-4ed8-8075-8e88dd1b51d4" containerName="istio-proxy" Nov 01 06:04:08.856895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.856880 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:08.906785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.906744 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 06:04:08.918712 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.918680 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:08.939969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.939939 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:08.974180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.974150 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5560761c-148d-4c7b-ad39-8f96eddcbd3d" path="/var/lib/kubelet/pods/5560761c-148d-4c7b-ad39-8f96eddcbd3d/volumes" Nov 01 06:04:08.974667 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:08.974651 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764e847b-5005-4bc2-b50f-af824b3d68a6" path="/var/lib/kubelet/pods/764e847b-5005-4bc2-b50f-af824b3d68a6/volumes" Nov 01 06:04:09.013244 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013262 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013356 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013508 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013492 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013532 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013598 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013559 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013592 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013720 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013702 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013742 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013810 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013866 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013844 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.013909 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013867 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013952 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013927 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.013992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013960 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kq4\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.014034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.013992 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfkv\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.014034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.014020 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.014125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.014064 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.014125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.014085 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.014321 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.014305 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 06:04:09.115563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115463 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.115563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115517 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.115563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115554 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.115868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115592 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.115868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115616 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.115868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115754 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.115868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115808 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.115868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115838 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kq4\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115881 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfkv\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115911 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115956 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.115982 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116032 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116054 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116064 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116087 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116139 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116185 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116399 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116415 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116495 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116548 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.116699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116613 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.117058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116713 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.117058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116499 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.117058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116697 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.117058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.116779 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.117453 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.117423 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.120123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.119364 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.120123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.119716 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.120966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.120945 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.121201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.121179 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.177606 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.177571 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.177768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.177616 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kq4\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-6x4ct\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.177768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.177662 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.177768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.177733 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfkv\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-cdk5z\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.454158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.454071 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:09.469965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.469933 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:09.904589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.904555 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:09.905150 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:04:09.905129 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef5cfe9_ca19_4396_a074_6e9e24c9a9b0.slice/crio-6051e52b43d502723c34083087eddfe49d3736b8e466bdd7569e17b03d20954d WatchSource:0}: Error finding container 6051e52b43d502723c34083087eddfe49d3736b8e466bdd7569e17b03d20954d: Status 404 returned error can't find the container with id 6051e52b43d502723c34083087eddfe49d3736b8e466bdd7569e17b03d20954d Nov 01 06:04:09.907791 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.907748 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:09.907926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.907828 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:09.907926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.907838 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:09.907926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.907890 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:09.907926 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:04:09.907842 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f7a1c5_9f51_4a37_9e0d_58d7e5870bae.slice/crio-3448e83f39ccc6020b125e289ee707d0e21f9ecb48e702289f5c20d9fab8f5f9 WatchSource:0}: Error finding container 3448e83f39ccc6020b125e289ee707d0e21f9ecb48e702289f5c20d9fab8f5f9: Status 404 returned error can't find the container with id 3448e83f39ccc6020b125e289ee707d0e21f9ecb48e702289f5c20d9fab8f5f9 Nov 01 06:04:09.910385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.910146 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:09.910385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.910214 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:09.910385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:09.910294 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:04:10.191872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.191765 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" event={"ID":"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae","Type":"ContainerStarted","Data":"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d"} Nov 01 06:04:10.191872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.191839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" event={"ID":"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae","Type":"ContainerStarted","Data":"3448e83f39ccc6020b125e289ee707d0e21f9ecb48e702289f5c20d9fab8f5f9"} Nov 01 06:04:10.193752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.193722 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" event={"ID":"def5cfe9-ca19-4396-a074-6e9e24c9a9b0","Type":"ContainerStarted","Data":"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e"} Nov 01 06:04:10.193939 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.193776 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" event={"ID":"def5cfe9-ca19-4396-a074-6e9e24c9a9b0","Type":"ContainerStarted","Data":"6051e52b43d502723c34083087eddfe49d3736b8e466bdd7569e17b03d20954d"} Nov 01 06:04:10.316135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.316069 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" podStartSLOduration=2.31605323 podStartE2EDuration="2.31605323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:10.313301557 +0000 UTC m=+3665.453890480" watchObservedRunningTime="2026-11-01 06:04:10.31605323 +0000 UTC m=+3665.456642148" Nov 01 06:04:10.437753 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.437691 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" podStartSLOduration=2.43767429 podStartE2EDuration="2.43767429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:10.435840845 +0000 UTC m=+3665.576429813" watchObservedRunningTime="2026-11-01 06:04:10.43767429 +0000 UTC m=+3665.578263208" Nov 01 06:04:10.476336 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.462429 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:10.476336 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.474414 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 01 06:04:10.476336 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.474490 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 01 06:04:10.483279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.479572 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:10.483279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.480989 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 01 06:04:10.483279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.481034 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 01 06:04:10.483890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.483857 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:10.489283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.488175 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 06:04:10.489283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.488402 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" containerID="cri-o://cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4" gracePeriod=30 Nov 01 06:04:10.497222 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.495506 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:10.512138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.512091 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:04:10.512528 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:10.512495 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" containerID="cri-o://ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32" gracePeriod=30 Nov 01 06:04:11.207644 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.207614 4425 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" secret="" err="secret \"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\" not found" Nov 01 06:04:11.208149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.207751 4425 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" secret="" err="secret \"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\" not found" Nov 01 06:04:11.458340 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.454722 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 01 06:04:11.458340 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.454787 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 01 06:04:11.477279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.474737 4425 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Nov 01 06:04:11.477279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:11.474818 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Nov 01 06:04:12.213959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:12.213911 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" containerID="cri-o://e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d" gracePeriod=30 Nov 01 06:04:12.214537 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:12.213959 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" containerID="cri-o://e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e" gracePeriod=30 Nov 01 06:04:13.907898 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:13.907815 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" containerID="cri-o://5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a" gracePeriod=25 Nov 01 06:04:14.015565 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:14.015520 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" containerID="cri-o://16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52" gracePeriod=25 Nov 01 06:04:15.934131 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.934102 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 06:04:15.936413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936391 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.936584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936570 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.936678 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936668 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.936773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936764 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936819 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936921 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.936953 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937055 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937131 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937168 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937177 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937182 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\") pod \"e8a31114-3554-4a28-b394-8ca65cd2684c\" (UID: \"e8a31114-3554-4a28-b394-8ca65cd2684c\") " Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937369 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:15.937623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937383 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:15.938191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937794 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:15.938191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937817 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:15.938191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937833 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:15.938191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937850 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:15.938191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.937864 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a31114-3554-4a28-b394-8ca65cd2684c-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:15.940007 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.939985 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:15.940831 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.940801 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:15.941037 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.940998 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "kube-api-access-5jcdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:15.941121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:15.941077 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy" pod "e8a31114-3554-4a28-b394-8ca65cd2684c" (UID: "e8a31114-3554-4a28-b394-8ca65cd2684c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.009966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.009936 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:04:16.039216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039188 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039266 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039307 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039338 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039388 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039410 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039453 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039493 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039523 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\") pod \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\" (UID: \"86c7bd23-7c17-4815-ae5d-e90de1e49ac8\") " Nov 01 06:04:16.039710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039610 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.039869 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039846 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.039931 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039905 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.039965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039922 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.039965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.039926 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:16.040080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040063 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040089 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040104 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040119 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcdq\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-kube-api-access-5jcdq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040134 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040377 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040148 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040377 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040162 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040377 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040178 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.040377 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.040192 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a31114-3554-4a28-b394-8ca65cd2684c-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.049450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.049398 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:16.050059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.050022 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:16.050213 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.050081 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "kube-api-access-djxhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:16.050213 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.050178 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token" pod "86c7bd23-7c17-4815-ae5d-e90de1e49ac8" (UID: "86c7bd23-7c17-4815-ae5d-e90de1e49ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:16.141310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.141238 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.141310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.141297 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.141310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.141314 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxhx\" (UniqueName: \"kubernetes.io/projected/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-kube-api-access-djxhx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.141618 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.141330 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c7bd23-7c17-4815-ae5d-e90de1e49ac8-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:16.244157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.244063 4425 generic.go:356] "Generic (PLEG): container finished" podID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerID="ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32" exitCode=0 Nov 01 06:04:16.244157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.244139 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" Nov 01 06:04:16.244434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.244161 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" event={"ID":"86c7bd23-7c17-4815-ae5d-e90de1e49ac8","Type":"ContainerDied","Data":"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32"} Nov 01 06:04:16.244434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.244241 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg" event={"ID":"86c7bd23-7c17-4815-ae5d-e90de1e49ac8","Type":"ContainerDied","Data":"1eed339a2a9a68715b428c62571ea635133d2f6fa644f4a9134f2664fcae8b1f"} Nov 01 06:04:16.244434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.244315 4425 scope.go:122] "RemoveContainer" containerID="ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32" Nov 01 06:04:16.246982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.246736 4425 generic.go:356] "Generic (PLEG): container finished" podID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerID="cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4" exitCode=0 Nov 01 06:04:16.246982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.246845 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" Nov 01 06:04:16.246982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.246882 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" event={"ID":"e8a31114-3554-4a28-b394-8ca65cd2684c","Type":"ContainerDied","Data":"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4"} Nov 01 06:04:16.246982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.246925 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj" event={"ID":"e8a31114-3554-4a28-b394-8ca65cd2684c","Type":"ContainerDied","Data":"eb88a783c7326d84ac08fc396cc38977521a97169db22c392b5696e32df8033c"} Nov 01 06:04:16.256622 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.256604 4425 scope.go:122] "RemoveContainer" containerID="ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32" Nov 01 06:04:16.257198 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:16.257081 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32\": container with ID starting with ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32 not found: ID does not exist" containerID="ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32" Nov 01 06:04:16.257198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.257117 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32"} err="failed to get container status \"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32\": rpc error: code = NotFound desc = could not find container \"ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32\": container with ID starting with ebbddcbf02786c7b4f0b42fabfef08977035fdedc3f5ea7546099bab2117fa32 not found: ID does not exist" Nov 01 06:04:16.257198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.257141 4425 scope.go:122] "RemoveContainer" containerID="cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4" Nov 01 06:04:16.270529 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.270505 4425 scope.go:122] "RemoveContainer" containerID="cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4" Nov 01 06:04:16.270916 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:16.270891 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4\": container with ID starting with cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4 not found: ID does not exist" containerID="cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4" Nov 01 06:04:16.271020 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.270930 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4"} err="failed to get container status \"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4\": rpc error: code = NotFound desc = could not find container \"cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4\": container with ID starting with cb7a263f331bdbcd866f9bee98e63bbc604b295839d4d2f2769c85f1088513e4 not found: ID does not exist" Nov 01 06:04:16.413480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.413414 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:04:16.417155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.417127 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-trkbg"] Nov 01 06:04:16.528216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.528126 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 06:04:16.533511 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.533480 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-j8mfj"] Nov 01 06:04:16.973809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.973762 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" path="/var/lib/kubelet/pods/86c7bd23-7c17-4815-ae5d-e90de1e49ac8/volumes" Nov 01 06:04:16.974376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:16.974283 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" path="/var/lib/kubelet/pods/e8a31114-3554-4a28-b394-8ca65cd2684c/volumes" Nov 01 06:04:18.047405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:18.047374 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb"] Nov 01 06:04:18.097914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:18.097882 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4aabf48-k76pb"] Nov 01 06:04:18.973416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:18.973380 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36397df4-3586-4802-a7aa-88d303e3927e" path="/var/lib/kubelet/pods/36397df4-3586-4802-a7aa-88d303e3927e/volumes" Nov 01 06:04:19.046320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:19.046291 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj"] Nov 01 06:04:19.097139 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:19.097114 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfca4d0-gvjkj"] Nov 01 06:04:20.975464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:20.975425 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f21dc0b-5ee6-4300-88c4-906e6ba44047" path="/var/lib/kubelet/pods/3f21dc0b-5ee6-4300-88c4-906e6ba44047/volumes" Nov 01 06:04:22.583581 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.583545 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:22.606818 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.606787 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.606981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.606898 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.606981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.606932 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.606981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.606973 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.607163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.607163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607046 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.607163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607083 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.607163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607127 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.607163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607138 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.607454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.607186 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\") pod \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\" (UID: \"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae\") " Nov 01 06:04:22.608460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.608215 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.609821 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.609713 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.609821 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.609796 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.611387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.610160 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:22.611387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.610373 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.611387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.610453 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.614117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.614091 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:22.614410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.614374 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "kube-api-access-p4kq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:22.614410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.614391 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo" pod "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" (UID: "16f7a1c5-9f51-4a37-9e0d-58d7e5870bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:22.671729 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.671695 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:22.709015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.708985 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709021 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709271 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709057 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709271 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709103 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709283 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709327 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709380 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.709408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709403 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709423 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.709631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709441 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709494 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\") pod \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\" (UID: \"def5cfe9-ca19-4396-a074-6e9e24c9a9b0\") " Nov 01 06:04:22.709804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.709804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709783 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:22.709986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709938 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.709986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709963 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.709986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709971 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.709979 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710017 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710035 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4kq4\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-kube-api-access-p4kq4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710052 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710067 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710080 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710092 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710106 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710119 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.710361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.710133 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.712117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.712095 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:22.712284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.712157 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:04:22.712284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.712214 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "kube-api-access-4xfkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:22.712862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.712837 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy" pod "def5cfe9-ca19-4396-a074-6e9e24c9a9b0" (UID: "def5cfe9-ca19-4396-a074-6e9e24c9a9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:04:22.810839 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.810807 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.810839 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.810837 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.811053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.810849 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xfkv\" (UniqueName: \"kubernetes.io/projected/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-kube-api-access-4xfkv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.811053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.810858 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:22.811053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:22.810867 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def5cfe9-ca19-4396-a074-6e9e24c9a9b0-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:23.028318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.028281 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:04:23.028318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.028311 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:04:23.296785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.296694 4425 generic.go:356] "Generic (PLEG): container finished" podID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerID="e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e" exitCode=0 Nov 01 06:04:23.296785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.296750 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" Nov 01 06:04:23.297022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.296779 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" event={"ID":"def5cfe9-ca19-4396-a074-6e9e24c9a9b0","Type":"ContainerDied","Data":"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e"} Nov 01 06:04:23.297022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.296837 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z" event={"ID":"def5cfe9-ca19-4396-a074-6e9e24c9a9b0","Type":"ContainerDied","Data":"6051e52b43d502723c34083087eddfe49d3736b8e466bdd7569e17b03d20954d"} Nov 01 06:04:23.297022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.296859 4425 scope.go:122] "RemoveContainer" containerID="e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e" Nov 01 06:04:23.298516 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.298492 4425 generic.go:356] "Generic (PLEG): container finished" podID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerID="e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d" exitCode=0 Nov 01 06:04:23.298623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.298532 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" Nov 01 06:04:23.298623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.298595 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" event={"ID":"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae","Type":"ContainerDied","Data":"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d"} Nov 01 06:04:23.298763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.298635 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct" event={"ID":"16f7a1c5-9f51-4a37-9e0d-58d7e5870bae","Type":"ContainerDied","Data":"3448e83f39ccc6020b125e289ee707d0e21f9ecb48e702289f5c20d9fab8f5f9"} Nov 01 06:04:23.309601 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.309582 4425 scope.go:122] "RemoveContainer" containerID="e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e" Nov 01 06:04:23.309930 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:23.309911 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e\": container with ID starting with e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e not found: ID does not exist" containerID="e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e" Nov 01 06:04:23.309989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.309938 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e"} err="failed to get container status \"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e\": rpc error: code = NotFound desc = could not find container \"e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e\": container with ID starting with e4bda8b94ec6c9636bc2145db8376c3e6f4df931f2dbdefdc8c8ace093e5b32e not found: ID does not exist" Nov 01 06:04:23.309989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.309958 4425 scope.go:122] "RemoveContainer" containerID="e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d" Nov 01 06:04:23.320295 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.320276 4425 scope.go:122] "RemoveContainer" containerID="e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d" Nov 01 06:04:23.320607 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:23.320580 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d\": container with ID starting with e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d not found: ID does not exist" containerID="e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d" Nov 01 06:04:23.320715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.320618 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d"} err="failed to get container status \"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d\": rpc error: code = NotFound desc = could not find container \"e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d\": container with ID starting with e1419cf801aef9127b7197bcd8dfc2e40a5c55efa213097d7dd1a151ecbcd07d not found: ID does not exist" Nov 01 06:04:23.465076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.465031 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:23.472294 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.472238 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-cdk5z"] Nov 01 06:04:23.579831 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.579801 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:23.584288 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:23.584244 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-6x4ct"] Nov 01 06:04:24.973384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:24.973348 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" path="/var/lib/kubelet/pods/16f7a1c5-9f51-4a37-9e0d-58d7e5870bae/volumes" Nov 01 06:04:24.973810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:24.973796 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" path="/var/lib/kubelet/pods/def5cfe9-ca19-4396-a074-6e9e24c9a9b0/volumes" Nov 01 06:04:26.948009 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:26.947970 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:04:33.027187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:33.027150 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:04:33.027187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:33.027176 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:04:34.159780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.159750 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 06:04:34.232594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.232565 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\") pod \"866fb760-d87d-4dbe-8673-a498b9fad4ee\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " Nov 01 06:04:34.232788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.232644 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\" (UniqueName: \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\") pod \"866fb760-d87d-4dbe-8673-a498b9fad4ee\" (UID: \"866fb760-d87d-4dbe-8673-a498b9fad4ee\") " Nov 01 06:04:34.233086 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.233050 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume" pod "866fb760-d87d-4dbe-8673-a498b9fad4ee" (UID: "866fb760-d87d-4dbe-8673-a498b9fad4ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:34.235850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.235822 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5" pod "866fb760-d87d-4dbe-8673-a498b9fad4ee" (UID: "866fb760-d87d-4dbe-8673-a498b9fad4ee"). InnerVolumeSpecName "kube-api-access-vwbq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:34.256664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.256639 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 06:04:34.334073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.334043 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\") pod \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " Nov 01 06:04:34.334273 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.334084 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\" (UniqueName: \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\") pod \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\" (UID: \"416a0576-85f9-414d-ab4a-1b1f4ed9dae6\") " Nov 01 06:04:34.334503 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.334484 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbq5\" (UniqueName: \"kubernetes.io/projected/866fb760-d87d-4dbe-8673-a498b9fad4ee-kube-api-access-vwbq5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:34.334543 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.334506 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume" pod "416a0576-85f9-414d-ab4a-1b1f4ed9dae6" (UID: "416a0576-85f9-414d-ab4a-1b1f4ed9dae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:34.334543 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.334512 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866fb760-d87d-4dbe-8673-a498b9fad4ee-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:34.336391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.336370 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w" pod "416a0576-85f9-414d-ab4a-1b1f4ed9dae6" (UID: "416a0576-85f9-414d-ab4a-1b1f4ed9dae6"). InnerVolumeSpecName "kube-api-access-jjl7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:34.367595 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.367560 4425 generic.go:356] "Generic (PLEG): container finished" podID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerID="16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52" exitCode=0 Nov 01 06:04:34.367763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.367608 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" Nov 01 06:04:34.367763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.367645 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" event={"ID":"416a0576-85f9-414d-ab4a-1b1f4ed9dae6","Type":"ContainerDied","Data":"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52"} Nov 01 06:04:34.367763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.367711 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g" event={"ID":"416a0576-85f9-414d-ab4a-1b1f4ed9dae6","Type":"ContainerDied","Data":"decc0541b6394b0002c9a0e606d9aa184ea4f9761d4e1d9dc7aa765670c0c5c0"} Nov 01 06:04:34.367763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.367746 4425 scope.go:122] "RemoveContainer" containerID="16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52" Nov 01 06:04:34.369111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.369083 4425 generic.go:356] "Generic (PLEG): container finished" podID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerID="5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a" exitCode=0 Nov 01 06:04:34.369219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.369124 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" Nov 01 06:04:34.369219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.369123 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" event={"ID":"866fb760-d87d-4dbe-8673-a498b9fad4ee","Type":"ContainerDied","Data":"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a"} Nov 01 06:04:34.369219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.369171 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28" event={"ID":"866fb760-d87d-4dbe-8673-a498b9fad4ee","Type":"ContainerDied","Data":"250a3634c068dc3da527d1fe599acf50c530e2a028f26e3f546d374805db3a33"} Nov 01 06:04:34.378482 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.378461 4425 scope.go:122] "RemoveContainer" containerID="16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52" Nov 01 06:04:34.378807 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:34.378783 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52\": container with ID starting with 16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52 not found: ID does not exist" containerID="16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52" Nov 01 06:04:34.378897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.378814 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52"} err="failed to get container status \"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52\": rpc error: code = NotFound desc = could not find container \"16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52\": container with ID starting with 16d357520d5726fdd592e2bfc11330bf7eb0eb813af642552b4c4baede25ae52 not found: ID does not exist" Nov 01 06:04:34.378897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.378833 4425 scope.go:122] "RemoveContainer" containerID="5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a" Nov 01 06:04:34.387597 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.387579 4425 scope.go:122] "RemoveContainer" containerID="5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a" Nov 01 06:04:34.387868 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:34.387850 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a\": container with ID starting with 5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a not found: ID does not exist" containerID="5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a" Nov 01 06:04:34.387906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.387877 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a"} err="failed to get container status \"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a\": rpc error: code = NotFound desc = could not find container \"5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a\": container with ID starting with 5d7d9ad753a6e2604d183bef103d4aec9b9d40aeeff871dd79f6196aff499f5a not found: ID does not exist" Nov 01 06:04:34.436159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.436132 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:34.436159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.436157 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjl7w\" (UniqueName: \"kubernetes.io/projected/416a0576-85f9-414d-ab4a-1b1f4ed9dae6-kube-api-access-jjl7w\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:34.509495 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.509409 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 06:04:34.513799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.513761 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1c4191-65f847c49f-dnq8g"] Nov 01 06:04:34.624217 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.624184 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 06:04:34.628426 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.628399 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1c4191-84cdfb95f7-qfl28"] Nov 01 06:04:34.972979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.972944 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" path="/var/lib/kubelet/pods/416a0576-85f9-414d-ab4a-1b1f4ed9dae6/volumes" Nov 01 06:04:34.973383 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:34.973370 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" path="/var/lib/kubelet/pods/866fb760-d87d-4dbe-8673-a498b9fad4ee/volumes" Nov 01 06:04:40.344359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344325 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw"] Nov 01 06:04:40.344852 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344837 4425 cpu_manager.go:414] "containerMap: removing container" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" Nov 01 06:04:40.344899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344854 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" Nov 01 06:04:40.344899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344869 4425 cpu_manager.go:414] "containerMap: removing container" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" Nov 01 06:04:40.344899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344877 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" Nov 01 06:04:40.344899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344890 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" Nov 01 06:04:40.344899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344896 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344909 4425 cpu_manager.go:414] "containerMap: removing container" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344913 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344932 4425 cpu_manager.go:414] "containerMap: removing container" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344937 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344951 4425 cpu_manager.go:414] "containerMap: removing container" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.344958 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345024 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="416a0576-85f9-414d-ab4a-1b1f4ed9dae6" containerName="payload-pre-processing" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345042 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c7bd23-7c17-4815-ae5d-e90de1e49ac8" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345052 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a31114-3554-4a28-b394-8ca65cd2684c" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345059 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="def5cfe9-ca19-4396-a074-6e9e24c9a9b0" containerName="istio-proxy" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345068 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="866fb760-d87d-4dbe-8673-a498b9fad4ee" containerName="payload-processing" Nov 01 06:04:40.345076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.345076 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f7a1c5-9f51-4a37-9e0d-58d7e5870bae" containerName="istio-proxy" Nov 01 06:04:40.348342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.348325 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.452797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.452768 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:04:40.452797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.452784 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:04:40.452797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.452769 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:04:40.453052 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.452871 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:04:40.455237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.455217 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw"] Nov 01 06:04:40.501363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.501328 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf496\" (UniqueName: \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.501540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.501460 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.501540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.501503 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.603068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.602983 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.603068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.603037 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.603307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.603112 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf496\" (UniqueName: \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.603897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.603809 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.605635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.605611 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.665034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.664999 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf496\" (UniqueName: \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\") pod \"maas-api-revoke-sub-36acd0e6-2hgdw\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:40.962478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:40.962376 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:41.339440 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:41.339416 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw"] Nov 01 06:04:41.341683 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:04:41.341654 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9577c589_e5d2_4bec_a4b0_eb7c18ae76cf.slice/crio-5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee WatchSource:0}: Error finding container 5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee: Status 404 returned error can't find the container with id 5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee Nov 01 06:04:41.444242 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:41.444210 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" event={"ID":"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf","Type":"ContainerStarted","Data":"5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee"} Nov 01 06:04:42.451626 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:42.451592 4425 generic.go:356] "Generic (PLEG): container finished" podID="9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" containerID="c36a60695c49dec585247e2548cc8148956f3c00ccef06f0d423c68dd5ea7379" exitCode=0 Nov 01 06:04:42.451998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:42.451671 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" event={"ID":"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf","Type":"ContainerDied","Data":"c36a60695c49dec585247e2548cc8148956f3c00ccef06f0d423c68dd5ea7379"} Nov 01 06:04:43.601225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.601200 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:43.736013 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.735921 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\") pod \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " Nov 01 06:04:43.736013 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.735981 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\") pod \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " Nov 01 06:04:43.736316 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.736071 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\" (UniqueName: \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\") pod \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\" (UID: \"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf\") " Nov 01 06:04:43.736395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.736324 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca" pod "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" (UID: "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:43.736559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.736542 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:43.738426 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.738398 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca" pod "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" (UID: "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:04:43.738686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.738667 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496" pod "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" (UID: "9577c589-e5d2-4bec-a4b0-eb7c18ae76cf"). InnerVolumeSpecName "kube-api-access-qf496". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:43.837383 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.837340 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:43.837383 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:43.837377 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf496\" (UniqueName: \"kubernetes.io/projected/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf-kube-api-access-qf496\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:44.464885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:44.464853 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" Nov 01 06:04:44.465086 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:44.464882 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw" event={"ID":"9577c589-e5d2-4bec-a4b0-eb7c18ae76cf","Type":"ContainerDied","Data":"5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee"} Nov 01 06:04:44.465086 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:44.464932 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eed8b3cc2f8160a675e6ce98ea6d9ea89eeb030d6796b2dfef815ad078157ee" Nov 01 06:04:44.709707 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:44.709670 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 06:04:44.720954 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:44.720888 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 06:04:52.735351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.735310 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv"] Nov 01 06:04:52.735982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.735812 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" containerName="revoke-keys" Nov 01 06:04:52.735982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.735824 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" containerName="revoke-keys" Nov 01 06:04:52.735982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.735923 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" containerName="revoke-keys" Nov 01 06:04:52.739114 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.739094 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.792914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.792885 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:04:52.793155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.792985 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:04:52.793308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.793045 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:04:52.796597 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.796574 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv"] Nov 01 06:04:52.825306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.825269 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.825464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.825350 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.825464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.825423 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kks9m\" (UniqueName: \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.843582 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.843555 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:04:52.926938 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.926903 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.927125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.926960 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kks9m\" (UniqueName: \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.927125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.927037 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.927581 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.927561 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.929508 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.929484 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:52.989182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:52.989106 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kks9m\" (UniqueName: \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\") pod \"maas-api-revoke-sub-b2a2803f-lc2bv\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:53.052501 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:53.052473 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:53.417877 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:53.417839 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 06:04:53.427270 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:53.427230 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv"] Nov 01 06:04:53.428791 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:04:53.428759 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb189d7e_39c8_4da5_8719_05d60bddbf3c.slice/crio-49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213 WatchSource:0}: Error finding container 49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213: Status 404 returned error can't find the container with id 49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213 Nov 01 06:04:53.524378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:53.524343 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" event={"ID":"cb189d7e-39c8-4da5-8719-05d60bddbf3c","Type":"ContainerStarted","Data":"49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213"} Nov 01 06:04:53.524702 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:53.524680 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 06:04:54.531410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:54.531373 4425 generic.go:356] "Generic (PLEG): container finished" podID="cb189d7e-39c8-4da5-8719-05d60bddbf3c" containerID="3edb8b83b44bc0d86bb87a04b9841157a4b1105e6b856eb6aa89cc76fcbe747e" exitCode=0 Nov 01 06:04:54.531806 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:54.531463 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" event={"ID":"cb189d7e-39c8-4da5-8719-05d60bddbf3c","Type":"ContainerDied","Data":"3edb8b83b44bc0d86bb87a04b9841157a4b1105e6b856eb6aa89cc76fcbe747e"} Nov 01 06:04:55.673285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.673225 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:55.769663 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.769632 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\") pod \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " Nov 01 06:04:55.769663 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.769672 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\" (UniqueName: \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\") pod \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " Nov 01 06:04:55.769942 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.769772 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\") pod \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\" (UID: \"cb189d7e-39c8-4da5-8719-05d60bddbf3c\") " Nov 01 06:04:55.770079 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.770056 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca" pod "cb189d7e-39c8-4da5-8719-05d60bddbf3c" (UID: "cb189d7e-39c8-4da5-8719-05d60bddbf3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:04:55.770193 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.770171 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:55.771971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.771939 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca" pod "cb189d7e-39c8-4da5-8719-05d60bddbf3c" (UID: "cb189d7e-39c8-4da5-8719-05d60bddbf3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:04:55.772101 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.771994 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m" pod "cb189d7e-39c8-4da5-8719-05d60bddbf3c" (UID: "cb189d7e-39c8-4da5-8719-05d60bddbf3c"). InnerVolumeSpecName "kube-api-access-kks9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:04:55.871796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.871766 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kks9m\" (UniqueName: \"kubernetes.io/projected/cb189d7e-39c8-4da5-8719-05d60bddbf3c-kube-api-access-kks9m\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:55.871796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:55.871794 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb189d7e-39c8-4da5-8719-05d60bddbf3c-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:04:56.544572 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:56.544540 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" Nov 01 06:04:56.544572 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:56.544556 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv" event={"ID":"cb189d7e-39c8-4da5-8719-05d60bddbf3c","Type":"ContainerDied","Data":"49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213"} Nov 01 06:04:56.544810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:56.544611 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d6ff81cd79cfd23f8dcb8358e888c79181449c87b5f8efef9d0d635ce64213" Nov 01 06:04:56.947549 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:04:56.947507 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:04:58.419155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:58.419094 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" containerID="cri-o://d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5" gracePeriod=25 Nov 01 06:04:58.525320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:58.525274 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" containerID="cri-o://aef9e2a1fa580b23f5d6d073272a1cfff34f2b73bc0d96e0debc3f5928dc794e" gracePeriod=25 Nov 01 06:04:59.730316 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:59.730274 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" containerID="cri-o://eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6" gracePeriod=45 Nov 01 06:04:59.734002 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:59.733972 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused" Nov 01 06:04:59.743834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:04:59.743802 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" containerID="cri-o://a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb" gracePeriod=45 Nov 01 06:05:00.018146 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.018115 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 06:05:00.040355 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.040329 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 06:05:00.119151 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119108 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\" (UniqueName: \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119210 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119312 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119364 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119389 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119438 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119489 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119516 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119563 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119595 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\") pod \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\" (UID: \"7a81a698-5cd0-4d60-9e57-71b6036dd7c0\") " Nov 01 06:05:00.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119658 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\" (UniqueName: \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119662 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119693 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\") pod \"7ded2e64-9d29-4f60-bf61-b99176c5721d\" (UID: \"7ded2e64-9d29-4f60-bf61-b99176c5721d\") " Nov 01 06:05:00.119907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119742 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.119949 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120026 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120311 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120332 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120345 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120357 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.120386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120366 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.120771 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.120424 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.122199 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.122159 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "kube-api-access-9tbxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:00.122524 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.122470 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:00.122833 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.122810 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:00.122983 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.122951 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm" pod "7a81a698-5cd0-4d60-9e57-71b6036dd7c0" (UID: "7a81a698-5cd0-4d60-9e57-71b6036dd7c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.123208 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.123182 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:00.123320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.123275 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2" pod "7ded2e64-9d29-4f60-bf61-b99176c5721d" (UID: "7ded2e64-9d29-4f60-bf61-b99176c5721d"). InnerVolumeSpecName "kube-api-access-drdl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:00.221674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221640 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221670 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221684 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdl2\" (UniqueName: \"kubernetes.io/projected/7ded2e64-9d29-4f60-bf61-b99176c5721d-kube-api-access-drdl2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221694 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded2e64-9d29-4f60-bf61-b99176c5721d-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221711 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbxk\" (UniqueName: \"kubernetes.io/projected/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-kube-api-access-9tbxk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221721 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221731 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a81a698-5cd0-4d60-9e57-71b6036dd7c0-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.221928 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.221739 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded2e64-9d29-4f60-bf61-b99176c5721d-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:00.571959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.571925 4425 generic.go:356] "Generic (PLEG): container finished" podID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerID="a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb" exitCode=0 Nov 01 06:05:00.572167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.571997 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" Nov 01 06:05:00.572167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.572018 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" event={"ID":"7a81a698-5cd0-4d60-9e57-71b6036dd7c0","Type":"ContainerDied","Data":"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb"} Nov 01 06:05:00.572167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.572060 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq" event={"ID":"7a81a698-5cd0-4d60-9e57-71b6036dd7c0","Type":"ContainerDied","Data":"129e0730f9a01ce2d618683233ce70e2f5f17079c6613fd9075af9a189262695"} Nov 01 06:05:00.572167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.572080 4425 scope.go:122] "RemoveContainer" containerID="a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb" Nov 01 06:05:00.573798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.573652 4425 generic.go:356] "Generic (PLEG): container finished" podID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerID="eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6" exitCode=0 Nov 01 06:05:00.573798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.573716 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" Nov 01 06:05:00.573798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.573745 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" event={"ID":"7ded2e64-9d29-4f60-bf61-b99176c5721d","Type":"ContainerDied","Data":"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6"} Nov 01 06:05:00.573798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.573774 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv" event={"ID":"7ded2e64-9d29-4f60-bf61-b99176c5721d","Type":"ContainerDied","Data":"1e1b426c344afb48c70cc2adca6617f28782c64d27a699a1eddb02feefb7b2a9"} Nov 01 06:05:00.586579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.586550 4425 scope.go:122] "RemoveContainer" containerID="a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb" Nov 01 06:05:00.591911 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:00.591866 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb\": container with ID starting with a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb not found: ID does not exist" containerID="a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb" Nov 01 06:05:00.592161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.592121 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb"} err="failed to get container status \"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb\": rpc error: code = NotFound desc = could not find container \"a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb\": container with ID starting with a130a6f7e251e85623ca6793d37950a216bf1d1411f126a30efbe6218f474adb not found: ID does not exist" Nov 01 06:05:00.592161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.592159 4425 scope.go:122] "RemoveContainer" containerID="eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6" Nov 01 06:05:00.605064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.605042 4425 scope.go:122] "RemoveContainer" containerID="eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6" Nov 01 06:05:00.605456 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:00.605431 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6\": container with ID starting with eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6 not found: ID does not exist" containerID="eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6" Nov 01 06:05:00.605545 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.605470 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6"} err="failed to get container status \"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6\": rpc error: code = NotFound desc = could not find container \"eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6\": container with ID starting with eea1e570c19bc6ad18b4e96424b00004f715ca3c464e1e950fcad02dc67cb9b6 not found: ID does not exist" Nov 01 06:05:00.743868 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.743830 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 06:05:00.790214 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.790177 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/premium-w0-7d1ec3-kserve-54777b9cff-c9hpv"] Nov 01 06:05:00.863332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.862301 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 06:05:00.868275 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:00.865596 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/simulator-w0-7d1ec3-kserve-56f5499cc4-9dhpq"] Nov 01 06:05:01.026244 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:01.026209 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" path="/var/lib/kubelet/pods/7a81a698-5cd0-4d60-9e57-71b6036dd7c0/volumes" Nov 01 06:05:01.027071 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:01.027047 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" path="/var/lib/kubelet/pods/7ded2e64-9d29-4f60-bf61-b99176c5721d/volumes" Nov 01 06:05:01.953781 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:01.953745 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 06:05:01.954313 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:01.954026 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" containerID="cri-o://83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5" gracePeriod=30 Nov 01 06:05:02.218038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.217938 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242"] Nov 01 06:05:02.218455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218438 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cb189d7e-39c8-4da5-8719-05d60bddbf3c" containerName="revoke-keys" Nov 01 06:05:02.218455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218455 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb189d7e-39c8-4da5-8719-05d60bddbf3c" containerName="revoke-keys" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218475 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218480 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218496 4425 cpu_manager.go:414] "containerMap: removing container" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218502 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218565 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a81a698-5cd0-4d60-9e57-71b6036dd7c0" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218574 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ded2e64-9d29-4f60-bf61-b99176c5721d" containerName="main" Nov 01 06:05:02.218636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.218585 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb189d7e-39c8-4da5-8719-05d60bddbf3c" containerName="revoke-keys" Nov 01 06:05:02.223234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.223209 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.277762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.277733 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:02.278036 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.277763 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:05:02.278165 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.277816 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:02.278165 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.277872 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:02.279658 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.279633 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242"] Nov 01 06:05:02.348723 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.348689 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.348926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.348760 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpd7\" (UniqueName: \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.348926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.348847 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.449690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.449649 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpd7\" (UniqueName: \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.449690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.449697 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.449946 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.449822 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.450434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.450409 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.452381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.452362 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.511892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.511810 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpd7\" (UniqueName: \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\") pod \"maas-api-revoke-sub-4a9d67e1-hw242\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.539700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.539670 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:02.916748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:02.916722 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242"] Nov 01 06:05:02.918452 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:05:02.918416 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6528f8_017a_4945_ba0a_ce00c7469a73.slice/crio-7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721 WatchSource:0}: Error finding container 7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721: Status 404 returned error can't find the container with id 7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721 Nov 01 06:05:03.525488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:03.525409 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-mpdzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:05:03.525488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:03.525462 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:05:03.596715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:03.596675 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" event={"ID":"dc6528f8-017a-4945-ba0a-ce00c7469a73","Type":"ContainerStarted","Data":"1b0bf470e6d80def9159527bfe71f3a2a4bb4a981fb12bc5e982d9c1f13c3370"} Nov 01 06:05:03.596916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:03.596733 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" event={"ID":"dc6528f8-017a-4945-ba0a-ce00c7469a73","Type":"ContainerStarted","Data":"7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721"} Nov 01 06:05:03.715449 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:03.715394 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" podStartSLOduration=1.715378577 podStartE2EDuration="1.715378577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:03.713791541 +0000 UTC m=+3718.854380460" watchObservedRunningTime="2026-11-01 06:05:03.715378577 +0000 UTC m=+3718.855967496" Nov 01 06:05:04.604889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:04.604852 4425 generic.go:356] "Generic (PLEG): container finished" podID="dc6528f8-017a-4945-ba0a-ce00c7469a73" containerID="1b0bf470e6d80def9159527bfe71f3a2a4bb4a981fb12bc5e982d9c1f13c3370" exitCode=0 Nov 01 06:05:04.605393 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:04.604950 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" event={"ID":"dc6528f8-017a-4945-ba0a-ce00c7469a73","Type":"ContainerDied","Data":"1b0bf470e6d80def9159527bfe71f3a2a4bb4a981fb12bc5e982d9c1f13c3370"} Nov 01 06:05:05.759296 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.759246 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:05.884897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.884820 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\") pod \"dc6528f8-017a-4945-ba0a-ce00c7469a73\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " Nov 01 06:05:05.885096 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.884894 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\") pod \"dc6528f8-017a-4945-ba0a-ce00c7469a73\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " Nov 01 06:05:05.885096 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.884935 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\" (UniqueName: \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\") pod \"dc6528f8-017a-4945-ba0a-ce00c7469a73\" (UID: \"dc6528f8-017a-4945-ba0a-ce00c7469a73\") " Nov 01 06:05:05.885293 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.885234 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca" pod "dc6528f8-017a-4945-ba0a-ce00c7469a73" (UID: "dc6528f8-017a-4945-ba0a-ce00c7469a73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:05.885419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.885394 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:05.887484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.887453 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7" pod "dc6528f8-017a-4945-ba0a-ce00c7469a73" (UID: "dc6528f8-017a-4945-ba0a-ce00c7469a73"). InnerVolumeSpecName "kube-api-access-vrpd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:05.887599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.887454 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca" pod "dc6528f8-017a-4945-ba0a-ce00c7469a73" (UID: "dc6528f8-017a-4945-ba0a-ce00c7469a73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:05.986564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.986526 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6528f8-017a-4945-ba0a-ce00c7469a73-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:05.986564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:05.986554 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpd7\" (UniqueName: \"kubernetes.io/projected/dc6528f8-017a-4945-ba0a-ce00c7469a73-kube-api-access-vrpd7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:06.192725 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:06.192627 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:05:06.192893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:06.192627 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:05:06.618908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:06.618874 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" Nov 01 06:05:06.619087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:06.618908 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242" event={"ID":"dc6528f8-017a-4945-ba0a-ce00c7469a73","Type":"ContainerDied","Data":"7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721"} Nov 01 06:05:06.619087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:06.618966 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4349fbe784179804fbc0e3e846b926fc0f4339ea2a60914b446fdfda69b721" Nov 01 06:05:07.223268 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.223225 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 06:05:07.302479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302396 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302437 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302486 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302535 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302559 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302616 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302644 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.302703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302671 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.303019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302766 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\") pod \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\" (UID: \"8d297bba-2e2e-4cad-9773-27cfb3b395f7\") " Nov 01 06:05:07.303019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302809 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:07.303019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302922 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:07.303019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302962 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:07.303019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.302984 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303040 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303233 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303289 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303305 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303320 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.303456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.303333 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.304892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.304862 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:05:07.305089 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.305072 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:07.305156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.305113 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "kube-api-access-859ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:07.305331 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.305316 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy" pod "8d297bba-2e2e-4cad-9773-27cfb3b395f7" (UID: "8d297bba-2e2e-4cad-9773-27cfb3b395f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:07.404722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.404690 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.404722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.404717 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.404722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.404726 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d297bba-2e2e-4cad-9773-27cfb3b395f7-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.404984 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.404736 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859ff\" (UniqueName: \"kubernetes.io/projected/8d297bba-2e2e-4cad-9773-27cfb3b395f7-kube-api-access-859ff\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:07.625773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.625732 4425 generic.go:356] "Generic (PLEG): container finished" podID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerID="83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5" exitCode=0 Nov 01 06:05:07.625933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.625778 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" Nov 01 06:05:07.625933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.625828 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" event={"ID":"8d297bba-2e2e-4cad-9773-27cfb3b395f7","Type":"ContainerDied","Data":"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5"} Nov 01 06:05:07.625933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.625881 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz" event={"ID":"8d297bba-2e2e-4cad-9773-27cfb3b395f7","Type":"ContainerDied","Data":"c11f95b89aecb4de27cffc063917dbb17c459107452b0200519aa13204aa5257"} Nov 01 06:05:07.625933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.625904 4425 scope.go:122] "RemoveContainer" containerID="83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5" Nov 01 06:05:07.636214 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.636195 4425 scope.go:122] "RemoveContainer" containerID="83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5" Nov 01 06:05:07.636505 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:07.636488 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5\": container with ID starting with 83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5 not found: ID does not exist" containerID="83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5" Nov 01 06:05:07.636559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.636513 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5"} err="failed to get container status \"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5\": rpc error: code = NotFound desc = could not find container \"83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5\": container with ID starting with 83a4ca54cb6a441eae7b1792ffe3c1a5fb95412bc78267405bcf010b0c356bc5 not found: ID does not exist" Nov 01 06:05:07.793030 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.792997 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 06:05:07.843743 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:07.843714 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-mpdzz"] Nov 01 06:05:08.095530 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.095500 4425 scope.go:122] "RemoveContainer" containerID="aa0787e8a8b14dec69b261f195bf95b0e245803c7e9dd3c45ac96a0a4c76e096" Nov 01 06:05:08.107079 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.107056 4425 scope.go:122] "RemoveContainer" containerID="74cd2823d7af6b8a24b8a870521578e252d02971515872d9fa69fe6cb752ba0a" Nov 01 06:05:08.116164 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.116142 4425 scope.go:122] "RemoveContainer" containerID="d61b2c3674ab5d00585eceabeffadab0910e64b5c03b2c91095fc5747a44622a" Nov 01 06:05:08.124215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.124198 4425 scope.go:122] "RemoveContainer" containerID="6e46b357a901f0589597ae0bd9dfdd5a667c630d960155ae7eee3a62f75acb58" Nov 01 06:05:08.132481 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.132464 4425 scope.go:122] "RemoveContainer" containerID="758e66859e06a16fd3bdaf51c8ad85f0c05a9a1c3939eab8efb7f021026aabc1" Nov 01 06:05:08.972854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:08.972822 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" path="/var/lib/kubelet/pods/8d297bba-2e2e-4cad-9773-27cfb3b395f7/volumes" Nov 01 06:05:12.916153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916117 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7"] Nov 01 06:05:12.916586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916573 4425 cpu_manager.go:414] "containerMap: removing container" podUID="dc6528f8-017a-4945-ba0a-ce00c7469a73" containerName="revoke-keys" Nov 01 06:05:12.916645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916587 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6528f8-017a-4945-ba0a-ce00c7469a73" containerName="revoke-keys" Nov 01 06:05:12.916645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916604 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" Nov 01 06:05:12.916645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916613 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" Nov 01 06:05:12.916744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916680 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6528f8-017a-4945-ba0a-ce00c7469a73" containerName="revoke-keys" Nov 01 06:05:12.916744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.916691 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d297bba-2e2e-4cad-9773-27cfb3b395f7" containerName="istio-proxy" Nov 01 06:05:12.919670 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.919654 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:12.974590 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.974555 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:12.974839 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:12.974619 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:13.025395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.025353 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:13.025594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.025478 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:05:13.027965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.027937 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7"] Nov 01 06:05:13.067025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.066987 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wq2x\" (UniqueName: \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.067230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.067050 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.067230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.067210 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.168825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.168726 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wq2x\" (UniqueName: \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.168825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.168811 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.169038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.168858 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.169520 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.169500 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.171458 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.171439 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.231462 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.231421 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wq2x\" (UniqueName: \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\") pod \"maas-api-revoke-sub-d67f334e-c68n7\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.232879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.232858 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:13.614861 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.614832 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7"] Nov 01 06:05:13.615881 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:05:13.615852 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729ff929_02f5_42b3_82f9_3afaca30f5b8.slice/crio-fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0 WatchSource:0}: Error finding container fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0: Status 404 returned error can't find the container with id fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0 Nov 01 06:05:13.661136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:13.661101 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" event={"ID":"729ff929-02f5-42b3-82f9-3afaca30f5b8","Type":"ContainerStarted","Data":"fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0"} Nov 01 06:05:14.671176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:14.671138 4425 generic.go:356] "Generic (PLEG): container finished" podID="729ff929-02f5-42b3-82f9-3afaca30f5b8" containerID="074d466772f6dc0be832e07ae4f4ba0cf9096daa9a525ca09253cc1bde1e50d4" exitCode=0 Nov 01 06:05:14.671623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:14.671227 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" event={"ID":"729ff929-02f5-42b3-82f9-3afaca30f5b8","Type":"ContainerDied","Data":"074d466772f6dc0be832e07ae4f4ba0cf9096daa9a525ca09253cc1bde1e50d4"} Nov 01 06:05:15.820438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.820411 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:15.994716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.994630 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\" (UniqueName: \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\") pod \"729ff929-02f5-42b3-82f9-3afaca30f5b8\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " Nov 01 06:05:15.994716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.994674 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\") pod \"729ff929-02f5-42b3-82f9-3afaca30f5b8\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " Nov 01 06:05:15.994984 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.994769 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\") pod \"729ff929-02f5-42b3-82f9-3afaca30f5b8\" (UID: \"729ff929-02f5-42b3-82f9-3afaca30f5b8\") " Nov 01 06:05:15.995276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.995228 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca" pod "729ff929-02f5-42b3-82f9-3afaca30f5b8" (UID: "729ff929-02f5-42b3-82f9-3afaca30f5b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:15.997065 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.997032 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca" pod "729ff929-02f5-42b3-82f9-3afaca30f5b8" (UID: "729ff929-02f5-42b3-82f9-3afaca30f5b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:15.997177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:15.997125 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x" pod "729ff929-02f5-42b3-82f9-3afaca30f5b8" (UID: "729ff929-02f5-42b3-82f9-3afaca30f5b8"). InnerVolumeSpecName "kube-api-access-7wq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:16.095998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.095944 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:16.095998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.095993 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wq2x\" (UniqueName: \"kubernetes.io/projected/729ff929-02f5-42b3-82f9-3afaca30f5b8-kube-api-access-7wq2x\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:16.095998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.096004 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729ff929-02f5-42b3-82f9-3afaca30f5b8-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:16.192524 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.192479 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:05:16.192524 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.192495 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:05:16.682898 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.682824 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" Nov 01 06:05:16.682898 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.682853 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7" event={"ID":"729ff929-02f5-42b3-82f9-3afaca30f5b8","Type":"ContainerDied","Data":"fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0"} Nov 01 06:05:16.683099 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:16.682910 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe7959d08e506bd16192d63e83f7398576f67b94965812423953ab4f9547ef0" Nov 01 06:05:18.679372 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.679302 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 06:05:18.698240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.697028 4425 generic.go:356] "Generic (PLEG): container finished" podID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerID="aef9e2a1fa580b23f5d6d073272a1cfff34f2b73bc0d96e0debc3f5928dc794e" exitCode=0 Nov 01 06:05:18.698240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.697240 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" event={"ID":"be6faddc-9c67-4a16-b812-2c69a3c82886","Type":"ContainerDied","Data":"aef9e2a1fa580b23f5d6d073272a1cfff34f2b73bc0d96e0debc3f5928dc794e"} Nov 01 06:05:18.699649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.699612 4425 generic.go:356] "Generic (PLEG): container finished" podID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerID="d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5" exitCode=0 Nov 01 06:05:18.699798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.699665 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" event={"ID":"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f","Type":"ContainerDied","Data":"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5"} Nov 01 06:05:18.699798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.699709 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" event={"ID":"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f","Type":"ContainerDied","Data":"faff8f113e800cb40d06c5fb5363136ef8fce33f5e653334bf7f26d2118798cf"} Nov 01 06:05:18.699798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.699740 4425 scope.go:122] "RemoveContainer" containerID="d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5" Nov 01 06:05:18.699999 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.699908 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v" Nov 01 06:05:18.712187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.712163 4425 scope.go:122] "RemoveContainer" containerID="d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5" Nov 01 06:05:18.712803 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:18.712749 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5\": container with ID starting with d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5 not found: ID does not exist" containerID="d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5" Nov 01 06:05:18.712922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.712796 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5"} err="failed to get container status \"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5\": rpc error: code = NotFound desc = could not find container \"d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5\": container with ID starting with d71a8fdd8400607cb1122eaaa3c19bdf33b6cd0fda62d53614c7d1fd3f8930d5 not found: ID does not exist" Nov 01 06:05:18.723297 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.723275 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\" (UniqueName: \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\") pod \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " Nov 01 06:05:18.723442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.723355 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\") pod \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\" (UID: \"cfb3bd52-fca2-41b1-9503-ad8e75da3e0f\") " Nov 01 06:05:18.723850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.723823 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume" pod "cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" (UID: "cfb3bd52-fca2-41b1-9503-ad8e75da3e0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:18.725599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.725577 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5" pod "cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" (UID: "cfb3bd52-fca2-41b1-9503-ad8e75da3e0f"). InnerVolumeSpecName "kube-api-access-pv4r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:18.766985 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.766957 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 06:05:18.824384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.824353 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\" (UniqueName: \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\") pod \"be6faddc-9c67-4a16-b812-2c69a3c82886\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " Nov 01 06:05:18.824585 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.824440 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\") pod \"be6faddc-9c67-4a16-b812-2c69a3c82886\" (UID: \"be6faddc-9c67-4a16-b812-2c69a3c82886\") " Nov 01 06:05:18.824688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.824674 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4r5\" (UniqueName: \"kubernetes.io/projected/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-kube-api-access-pv4r5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:18.824760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.824691 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:18.824872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.824849 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume" pod "be6faddc-9c67-4a16-b812-2c69a3c82886" (UID: "be6faddc-9c67-4a16-b812-2c69a3c82886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:18.826674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.826654 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6" pod "be6faddc-9c67-4a16-b812-2c69a3c82886" (UID: "be6faddc-9c67-4a16-b812-2c69a3c82886"). InnerVolumeSpecName "kube-api-access-xhlh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:18.925312 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.925278 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6faddc-9c67-4a16-b812-2c69a3c82886-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:18.925312 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:18.925304 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlh6\" (UniqueName: \"kubernetes.io/projected/be6faddc-9c67-4a16-b812-2c69a3c82886-kube-api-access-xhlh6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:19.118470 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.118406 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 06:05:19.123589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.123551 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7d1ec3-5c5c8b5577-5z46v"] Nov 01 06:05:19.706158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.706077 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" Nov 01 06:05:19.706158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.706086 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg" event={"ID":"be6faddc-9c67-4a16-b812-2c69a3c82886","Type":"ContainerDied","Data":"ec3c32be6ff817f0bfc011937d527affab76ddbb697bc0ff6553f4cd16747d3c"} Nov 01 06:05:19.706772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.706170 4425 scope.go:122] "RemoveContainer" containerID="aef9e2a1fa580b23f5d6d073272a1cfff34f2b73bc0d96e0debc3f5928dc794e" Nov 01 06:05:19.873308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.873276 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 06:05:19.879320 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:19.879294 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7d1ec3-6b66456595-2fczg"] Nov 01 06:05:20.972557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:20.972520 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" path="/var/lib/kubelet/pods/be6faddc-9c67-4a16-b812-2c69a3c82886/volumes" Nov 01 06:05:20.972961 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:20.972905 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" path="/var/lib/kubelet/pods/cfb3bd52-fca2-41b1-9503-ad8e75da3e0f/volumes" Nov 01 06:05:21.978966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.978934 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:21.979419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979357 4425 cpu_manager.go:414] "containerMap: removing container" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" Nov 01 06:05:21.979419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979369 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" Nov 01 06:05:21.979419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979403 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" Nov 01 06:05:21.979419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979408 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" Nov 01 06:05:21.979419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979419 4425 cpu_manager.go:414] "containerMap: removing container" podUID="729ff929-02f5-42b3-82f9-3afaca30f5b8" containerName="revoke-keys" Nov 01 06:05:21.979609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979424 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729ff929-02f5-42b3-82f9-3afaca30f5b8" containerName="revoke-keys" Nov 01 06:05:21.979609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979490 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb3bd52-fca2-41b1-9503-ad8e75da3e0f" containerName="payload-processing" Nov 01 06:05:21.979609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979499 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="729ff929-02f5-42b3-82f9-3afaca30f5b8" containerName="revoke-keys" Nov 01 06:05:21.979609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.979507 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6faddc-9c67-4a16-b812-2c69a3c82886" containerName="payload-pre-processing" Nov 01 06:05:21.984147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:21.984123 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.052200 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.052172 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\"" Nov 01 06:05:22.056628 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056601 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056679 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056704 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056728 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvqv\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056817 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056866 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.056969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056889 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.057007 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.056975 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.057056 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.057033 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.100834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.100797 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:22.158072 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158032 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158080 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158111 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158141 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvqv\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158169 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158195 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158221 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158328 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158411 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158629 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158760 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158835 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.158978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.158887 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.159057 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.159041 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.161105 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.161081 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.161230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.161105 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.221691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.221130 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvqv\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.221691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.221143 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.297314 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.297228 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:22.726923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.726897 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:22.728828 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:05:22.728803 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70157360_959c_4223_ba94_12e5754b2d2d.slice/crio-a29823128b2371daa5966945622f16dbd433c0be8476cd07bf04f000b617d006 WatchSource:0}: Error finding container a29823128b2371daa5966945622f16dbd433c0be8476cd07bf04f000b617d006: Status 404 returned error can't find the container with id a29823128b2371daa5966945622f16dbd433c0be8476cd07bf04f000b617d006 Nov 01 06:05:22.731002 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.730972 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:05:22.731060 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.731036 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:05:22.731108 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:22.731067 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:05:23.731469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:23.731427 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" event={"ID":"70157360-959c-4223-ba94-12e5754b2d2d","Type":"ContainerStarted","Data":"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1"} Nov 01 06:05:23.731893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:23.731530 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" event={"ID":"70157360-959c-4223-ba94-12e5754b2d2d","Type":"ContainerStarted","Data":"a29823128b2371daa5966945622f16dbd433c0be8476cd07bf04f000b617d006"} Nov 01 06:05:23.906936 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:23.906879 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" podStartSLOduration=2.906861994 podStartE2EDuration="2.906861994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:23.904074235 +0000 UTC m=+3739.044663159" watchObservedRunningTime="2026-11-01 06:05:23.906861994 +0000 UTC m=+3739.047450912" Nov 01 06:05:24.297602 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.297567 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:24.299189 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.299157 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 01 06:05:24.299361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.299210 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 01 06:05:24.708701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.708661 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn"] Nov 01 06:05:24.712638 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.712616 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.785239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.785200 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzf4\" (UniqueName: \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.785754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.785397 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.785754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.785468 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.817549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.817514 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:24.817549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.817514 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:24.817799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.817687 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:24.817799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.817757 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:05:24.820164 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.820144 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn"] Nov 01 06:05:24.887022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.886993 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzf4\" (UniqueName: \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.887227 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.887093 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.887227 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.887141 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.887925 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.887906 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.889755 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.889738 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:24.969555 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:24.969480 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzf4\" (UniqueName: \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\") pod \"maas-api-revoke-sub-e16b75f1-zs4kn\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:25.026182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.026146 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:25.298545 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.298460 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 01 06:05:25.298545 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.298517 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 01 06:05:25.431878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.431800 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn"] Nov 01 06:05:25.433122 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:05:25.432700 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fedf85_f5bf_4d05_8e0b_765999948026.slice/crio-3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072 WatchSource:0}: Error finding container 3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072: Status 404 returned error can't find the container with id 3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072 Nov 01 06:05:25.733861 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.733819 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:25.775335 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.775290 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" event={"ID":"13fedf85-f5bf-4d05-8e0b-765999948026","Type":"ContainerStarted","Data":"3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072"} Nov 01 06:05:25.775676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:25.775657 4425 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" secret="" err="secret \"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\" not found" Nov 01 06:05:26.298521 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:26.298487 4425 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 01 06:05:26.298941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:26.298541 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 01 06:05:26.782201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:26.782169 4425 generic.go:356] "Generic (PLEG): container finished" podID="13fedf85-f5bf-4d05-8e0b-765999948026" containerID="c01efd8140006272e87f112d8bfb92d8b16252d4b4862c12dbe0dc8bcb7c39f1" exitCode=0 Nov 01 06:05:26.782403 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:26.782282 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" event={"ID":"13fedf85-f5bf-4d05-8e0b-765999948026","Type":"ContainerDied","Data":"c01efd8140006272e87f112d8bfb92d8b16252d4b4862c12dbe0dc8bcb7c39f1"} Nov 01 06:05:26.782493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:26.782472 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" containerID="cri-o://a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1" gracePeriod=30 Nov 01 06:05:26.971937 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:26.971900 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:05:27.978485 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:27.978453 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:28.019954 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.019924 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\") pod \"13fedf85-f5bf-4d05-8e0b-765999948026\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " Nov 01 06:05:28.020121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.019974 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\") pod \"13fedf85-f5bf-4d05-8e0b-765999948026\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " Nov 01 06:05:28.020121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.020008 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\" (UniqueName: \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\") pod \"13fedf85-f5bf-4d05-8e0b-765999948026\" (UID: \"13fedf85-f5bf-4d05-8e0b-765999948026\") " Nov 01 06:05:28.020470 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.020437 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca" pod "13fedf85-f5bf-4d05-8e0b-765999948026" (UID: "13fedf85-f5bf-4d05-8e0b-765999948026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:28.020632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.020581 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:28.022446 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.022421 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca" pod "13fedf85-f5bf-4d05-8e0b-765999948026" (UID: "13fedf85-f5bf-4d05-8e0b-765999948026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:28.022446 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.022440 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4" pod "13fedf85-f5bf-4d05-8e0b-765999948026" (UID: "13fedf85-f5bf-4d05-8e0b-765999948026"). InnerVolumeSpecName "kube-api-access-fjzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:28.121684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.121649 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzf4\" (UniqueName: \"kubernetes.io/projected/13fedf85-f5bf-4d05-8e0b-765999948026-kube-api-access-fjzf4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:28.121684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.121681 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13fedf85-f5bf-4d05-8e0b-765999948026-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:28.794155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.794117 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" event={"ID":"13fedf85-f5bf-4d05-8e0b-765999948026","Type":"ContainerDied","Data":"3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072"} Nov 01 06:05:28.794399 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.794203 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3726add4bd60ddbf2a4fa5435e666f937f96e34a13645968dbbfb4fa3ec53072" Nov 01 06:05:28.794399 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:28.794216 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn" Nov 01 06:05:34.547789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.547760 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:34.692097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692006 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692095 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692129 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692174 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692207 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692239 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692322 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:34.692404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692349 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692417 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692471 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\") pod \"70157360-959c-4223-ba94-12e5754b2d2d\" (UID: \"70157360-959c-4223-ba94-12e5754b2d2d\") " Nov 01 06:05:34.692766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692475 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:34.692766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692522 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:34.692766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692674 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:34.693084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692918 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:34.693084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.692924 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.693428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.693132 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.693428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.693158 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.693428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.693174 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.694673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.694642 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:05:34.694673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.694663 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:05:34.694844 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.694670 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:34.694844 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.694836 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv" pod "70157360-959c-4223-ba94-12e5754b2d2d" (UID: "70157360-959c-4223-ba94-12e5754b2d2d"). InnerVolumeSpecName "kube-api-access-spvqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:34.793885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.793844 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70157360-959c-4223-ba94-12e5754b2d2d-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.793885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.793874 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70157360-959c-4223-ba94-12e5754b2d2d-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.793885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.793884 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.794148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.793914 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvqv\" (UniqueName: \"kubernetes.io/projected/70157360-959c-4223-ba94-12e5754b2d2d-kube-api-access-spvqv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.794148 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.793925 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70157360-959c-4223-ba94-12e5754b2d2d-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:34.828790 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.828754 4425 generic.go:356] "Generic (PLEG): container finished" podID="70157360-959c-4223-ba94-12e5754b2d2d" containerID="a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1" exitCode=0 Nov 01 06:05:34.828981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.828817 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" Nov 01 06:05:34.828981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.828839 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" event={"ID":"70157360-959c-4223-ba94-12e5754b2d2d","Type":"ContainerDied","Data":"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1"} Nov 01 06:05:34.828981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.828932 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl" event={"ID":"70157360-959c-4223-ba94-12e5754b2d2d","Type":"ContainerDied","Data":"a29823128b2371daa5966945622f16dbd433c0be8476cd07bf04f000b617d006"} Nov 01 06:05:34.828981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.828962 4425 scope.go:122] "RemoveContainer" containerID="a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1" Nov 01 06:05:34.838995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.838975 4425 scope.go:122] "RemoveContainer" containerID="a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1" Nov 01 06:05:34.839299 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:34.839242 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1\": container with ID starting with a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1 not found: ID does not exist" containerID="a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1" Nov 01 06:05:34.839358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.839314 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1"} err="failed to get container status \"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1\": rpc error: code = NotFound desc = could not find container \"a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1\": container with ID starting with a944d95b46ab93587115102884df6ddafb36313cece55828420b991e43ff94a1 not found: ID does not exist" Nov 01 06:05:34.996989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:34.996908 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:35.002664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.002636 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-kdkzl"] Nov 01 06:05:35.545691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.545659 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp"] Nov 01 06:05:35.546097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546083 4425 cpu_manager.go:414] "containerMap: removing container" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" Nov 01 06:05:35.546097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546096 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" Nov 01 06:05:35.546221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546131 4425 cpu_manager.go:414] "containerMap: removing container" podUID="13fedf85-f5bf-4d05-8e0b-765999948026" containerName="revoke-keys" Nov 01 06:05:35.546221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546137 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fedf85-f5bf-4d05-8e0b-765999948026" containerName="revoke-keys" Nov 01 06:05:35.546221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546202 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fedf85-f5bf-4d05-8e0b-765999948026" containerName="revoke-keys" Nov 01 06:05:35.546221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.546213 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="70157360-959c-4223-ba94-12e5754b2d2d" containerName="istio-proxy" Nov 01 06:05:35.551776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.551758 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.601786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.601753 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.601958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.601894 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s7v\" (UniqueName: \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.602008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.601985 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.605549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.605523 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:05:35.605718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.605561 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:05:35.607944 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.607920 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp"] Nov 01 06:05:35.654978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.654950 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:05:35.654978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.654950 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:05:35.702810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.702776 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.703023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.702825 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.703023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.702918 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s7v\" (UniqueName: \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.703427 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.703403 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.705404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.705381 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.764785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.764746 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s7v\" (UniqueName: \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\") pod \"maas-api-revoke-sub-f638148b-v6gjp\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:35.864752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:35.864715 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:36.239615 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:36.239584 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp"] Nov 01 06:05:36.241085 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:05:36.241057 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c8a409_df69_4152_9a60_b57d45130ace.slice/crio-378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9 WatchSource:0}: Error finding container 378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9: Status 404 returned error can't find the container with id 378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9 Nov 01 06:05:36.844395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:36.844360 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" event={"ID":"d6c8a409-df69-4152-9a60-b57d45130ace","Type":"ContainerStarted","Data":"c83244fada4333eca9d567ad8e03c4229a5c635d7e31820add4a2e004a5e4e68"} Nov 01 06:05:36.844801 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:36.844415 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" event={"ID":"d6c8a409-df69-4152-9a60-b57d45130ace","Type":"ContainerStarted","Data":"378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9"} Nov 01 06:05:36.962374 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:36.962313 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" podStartSLOduration=1.9622945939999998 podStartE2EDuration="1.962294594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:36.96079313 +0000 UTC m=+3752.101382051" watchObservedRunningTime="2026-11-01 06:05:36.962294594 +0000 UTC m=+3752.102883514" Nov 01 06:05:36.972831 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:36.972802 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70157360-959c-4223-ba94-12e5754b2d2d" path="/var/lib/kubelet/pods/70157360-959c-4223-ba94-12e5754b2d2d/volumes" Nov 01 06:05:37.851163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:37.851132 4425 generic.go:356] "Generic (PLEG): container finished" podID="d6c8a409-df69-4152-9a60-b57d45130ace" containerID="c83244fada4333eca9d567ad8e03c4229a5c635d7e31820add4a2e004a5e4e68" exitCode=0 Nov 01 06:05:37.851627 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:37.851214 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" event={"ID":"d6c8a409-df69-4152-9a60-b57d45130ace","Type":"ContainerDied","Data":"c83244fada4333eca9d567ad8e03c4229a5c635d7e31820add4a2e004a5e4e68"} Nov 01 06:05:39.001131 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.001103 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:39.141488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.141412 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\" (UniqueName: \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\") pod \"d6c8a409-df69-4152-9a60-b57d45130ace\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " Nov 01 06:05:39.141656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.141489 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\") pod \"d6c8a409-df69-4152-9a60-b57d45130ace\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " Nov 01 06:05:39.141656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.141563 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\") pod \"d6c8a409-df69-4152-9a60-b57d45130ace\" (UID: \"d6c8a409-df69-4152-9a60-b57d45130ace\") " Nov 01 06:05:39.141977 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.141951 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca" pod "d6c8a409-df69-4152-9a60-b57d45130ace" (UID: "d6c8a409-df69-4152-9a60-b57d45130ace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:05:39.143764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.143737 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v" pod "d6c8a409-df69-4152-9a60-b57d45130ace" (UID: "d6c8a409-df69-4152-9a60-b57d45130ace"). InnerVolumeSpecName "kube-api-access-p9s7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:05:39.143904 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.143817 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca" pod "d6c8a409-df69-4152-9a60-b57d45130ace" (UID: "d6c8a409-df69-4152-9a60-b57d45130ace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:05:39.243102 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.243067 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:39.243102 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.243098 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s7v\" (UniqueName: \"kubernetes.io/projected/d6c8a409-df69-4152-9a60-b57d45130ace-kube-api-access-p9s7v\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:39.243102 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.243108 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c8a409-df69-4152-9a60-b57d45130ace-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:05:39.863330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.863288 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" event={"ID":"d6c8a409-df69-4152-9a60-b57d45130ace","Type":"ContainerDied","Data":"378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9"} Nov 01 06:05:39.863330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.863315 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp" Nov 01 06:05:39.863592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:39.863381 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378af75f17f71565135b1bf9572955a8f1e967545d4b579c31110eda8887b0d9" Nov 01 06:05:46.047649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:46.047608 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2"] Nov 01 06:05:46.098594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:46.098557 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0882d3-dzms2"] Nov 01 06:05:46.973025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:46.972990 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1742b46-1376-4463-9cba-23eda1ce82a8" path="/var/lib/kubelet/pods/f1742b46-1376-4463-9cba-23eda1ce82a8/volumes" Nov 01 06:05:56.046646 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:56.046612 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27"] Nov 01 06:05:56.097351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:56.097314 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd18e3c-6kj27"] Nov 01 06:05:56.946211 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:05:56.946167 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:05:56.972086 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:05:56.972050 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d000678f-900b-4cc3-9fbc-e3a66511241b" path="/var/lib/kubelet/pods/d000678f-900b-4cc3-9fbc-e3a66511241b/volumes" Nov 01 06:06:08.203401 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:08.203369 4425 scope.go:122] "RemoveContainer" containerID="09182654c540a61c598a353594841efdbcb59d80f8b4df45561d7f1820bc9515" Nov 01 06:06:08.212627 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:08.212604 4425 scope.go:122] "RemoveContainer" containerID="173b809eaa537c5c5590dfe03aaa5bff981c7600b4320cb3ec70c521b35a0c88" Nov 01 06:06:23.047124 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:23.047092 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn"] Nov 01 06:06:23.097595 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:23.097562 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e238df4a-m7lbn"] Nov 01 06:06:24.972962 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:24.972919 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268f6f37-4e71-4537-a5a7-66ad18986f25" path="/var/lib/kubelet/pods/268f6f37-4e71-4537-a5a7-66ad18986f25/volumes" Nov 01 06:06:26.947179 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:26.947142 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:06:37.046812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:37.046772 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw"] Nov 01 06:06:37.104509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:37.099394 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a88d1a54-m9ssw"] Nov 01 06:06:38.456087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.456056 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7443-0b0e22"] Nov 01 06:06:38.456560 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.456543 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d6c8a409-df69-4152-9a60-b57d45130ace" containerName="revoke-keys" Nov 01 06:06:38.456607 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.456562 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c8a409-df69-4152-9a60-b57d45130ace" containerName="revoke-keys" Nov 01 06:06:38.456678 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.456667 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c8a409-df69-4152-9a60-b57d45130ace" containerName="revoke-keys" Nov 01 06:06:38.460180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.460163 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:38.512561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.512530 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzwg\" (UniqueName: \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\") pod \"test-curl-7443-0b0e22\" (UID: \"8dc6cc77-8238-402b-9ffe-fdbbef8518bb\") " pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:38.515060 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.515031 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7443-0b0e22"] Nov 01 06:06:38.562987 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.562948 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wfzxk\"" Nov 01 06:06:38.613549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.613516 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzwg\" (UniqueName: \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\") pod \"test-curl-7443-0b0e22\" (UID: \"8dc6cc77-8238-402b-9ffe-fdbbef8518bb\") " pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:38.675688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.675652 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzwg\" (UniqueName: \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\") pod \"test-curl-7443-0b0e22\" (UID: \"8dc6cc77-8238-402b-9ffe-fdbbef8518bb\") " pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:38.771629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.771527 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:38.973487 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:38.973454 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d2e15f-dfbd-452b-a875-945270cc73ae" path="/var/lib/kubelet/pods/d8d2e15f-dfbd-452b-a875-945270cc73ae/volumes" Nov 01 06:06:39.143431 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:39.143401 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7443-0b0e22"] Nov 01 06:06:39.144837 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:06:39.144810 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc6cc77_8238_402b_9ffe_fdbbef8518bb.slice/crio-ef6243bba7d05151dc8f12e6e15bd2560901e1d20abb0b6e68e46166939553c4 WatchSource:0}: Error finding container ef6243bba7d05151dc8f12e6e15bd2560901e1d20abb0b6e68e46166939553c4: Status 404 returned error can't find the container with id ef6243bba7d05151dc8f12e6e15bd2560901e1d20abb0b6e68e46166939553c4 Nov 01 06:06:39.195474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:39.195448 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7443-0b0e22" event={"ID":"8dc6cc77-8238-402b-9ffe-fdbbef8518bb","Type":"ContainerStarted","Data":"ef6243bba7d05151dc8f12e6e15bd2560901e1d20abb0b6e68e46166939553c4"} Nov 01 06:06:40.203014 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:40.202975 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7443-0b0e22" event={"ID":"8dc6cc77-8238-402b-9ffe-fdbbef8518bb","Type":"ContainerStarted","Data":"2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d"} Nov 01 06:06:40.317532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:40.317477 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7443-0b0e22" podStartSLOduration=2.317463869 podStartE2EDuration="2.317463869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:40.317019185 +0000 UTC m=+3815.457608127" watchObservedRunningTime="2026-11-01 06:06:40.317463869 +0000 UTC m=+3815.458052787" Nov 01 06:06:41.804980 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.804948 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7443-0b0e22"] Nov 01 06:06:41.808989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.808962 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7443-0b0e22"] Nov 01 06:06:41.960018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.959985 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7443-5cc14f"] Nov 01 06:06:41.960479 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.960464 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" containerName="test-curl-7443-0b0e22" Nov 01 06:06:41.960540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.960481 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" containerName="test-curl-7443-0b0e22" Nov 01 06:06:41.960582 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.960549 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" containerName="test-curl-7443-0b0e22" Nov 01 06:06:41.963797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:41.963779 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:42.020032 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.019997 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7443-5cc14f"] Nov 01 06:06:42.045812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.045772 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbp2k\" (UniqueName: \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\") pod \"test-curl-7443-5cc14f\" (UID: \"042d825e-d6d1-48b8-b386-a3cbb8f6127c\") " pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:42.147378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.147345 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbp2k\" (UniqueName: \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\") pod \"test-curl-7443-5cc14f\" (UID: \"042d825e-d6d1-48b8-b386-a3cbb8f6127c\") " pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:42.209570 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.209536 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbp2k\" (UniqueName: \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\") pod \"test-curl-7443-5cc14f\" (UID: \"042d825e-d6d1-48b8-b386-a3cbb8f6127c\") " pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:42.213215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.213181 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7443-0b0e22" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" containerName="test-curl-7443-0b0e22" containerID="cri-o://2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d" gracePeriod=2 Nov 01 06:06:42.265306 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:42.265246 4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7443-0b0e22\" is forbidden: User \"system:node:ip-10-0-141-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-97.us-west-1.compute.internal' and this object" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:42.275371 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.275346 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:42.647593 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:42.647569 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7443-5cc14f"] Nov 01 06:06:42.649054 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:06:42.649028 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042d825e_d6d1_48b8_b386_a3cbb8f6127c.slice/crio-47a4026abca9be4ac39076ddfc0e7550e36e114eebc345530a386af568454a78 WatchSource:0}: Error finding container 47a4026abca9be4ac39076ddfc0e7550e36e114eebc345530a386af568454a78: Status 404 returned error can't find the container with id 47a4026abca9be4ac39076ddfc0e7550e36e114eebc345530a386af568454a78 Nov 01 06:06:43.220189 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:43.220152 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7443-5cc14f" event={"ID":"042d825e-d6d1-48b8-b386-a3cbb8f6127c","Type":"ContainerStarted","Data":"1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f"} Nov 01 06:06:43.220630 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:43.220201 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7443-5cc14f" event={"ID":"042d825e-d6d1-48b8-b386-a3cbb8f6127c","Type":"ContainerStarted","Data":"47a4026abca9be4ac39076ddfc0e7550e36e114eebc345530a386af568454a78"} Nov 01 06:06:43.336151 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:43.336092 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7443-5cc14f" podStartSLOduration=2.336078967 podStartE2EDuration="2.336078967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:43.33500378 +0000 UTC m=+3818.475592698" watchObservedRunningTime="2026-11-01 06:06:43.336078967 +0000 UTC m=+3818.476667922" Nov 01 06:06:44.445471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.445436 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:44.468378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.468303 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\" (UniqueName: \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\") pod \"8dc6cc77-8238-402b-9ffe-fdbbef8518bb\" (UID: \"8dc6cc77-8238-402b-9ffe-fdbbef8518bb\") " Nov 01 06:06:44.470550 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.470526 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg" pod "8dc6cc77-8238-402b-9ffe-fdbbef8518bb" (UID: "8dc6cc77-8238-402b-9ffe-fdbbef8518bb"). InnerVolumeSpecName "kube-api-access-nfzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:44.569773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.569741 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfzwg\" (UniqueName: \"kubernetes.io/projected/8dc6cc77-8238-402b-9ffe-fdbbef8518bb-kube-api-access-nfzwg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:44.768932 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.768834 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7443-5cc14f"] Nov 01 06:06:44.773391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.773362 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7443-5cc14f"] Nov 01 06:06:44.972368 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:44.972337 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" path="/var/lib/kubelet/pods/8dc6cc77-8238-402b-9ffe-fdbbef8518bb/volumes" Nov 01 06:06:45.231515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.231483 4425 generic.go:356] "Generic (PLEG): container finished" podID="8dc6cc77-8238-402b-9ffe-fdbbef8518bb" containerID="2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d" exitCode=137 Nov 01 06:06:45.231716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.231521 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-0b0e22" Nov 01 06:06:45.231716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.231580 4425 scope.go:122] "RemoveContainer" containerID="2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d" Nov 01 06:06:45.231716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.231701 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7443-5cc14f" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" containerName="test-curl-7443-5cc14f" containerID="cri-o://1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f" gracePeriod=2 Nov 01 06:06:45.240430 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.240410 4425 scope.go:122] "RemoveContainer" containerID="2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d" Nov 01 06:06:45.240697 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:45.240673 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d\": container with ID starting with 2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d not found: ID does not exist" containerID="2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d" Nov 01 06:06:45.240777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.240711 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d"} err="failed to get container status \"2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d\": rpc error: code = NotFound desc = could not find container \"2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d\": container with ID starting with 2b7856c4c86b95e8205286d999999d5dd2f5d4fdb23572f962f5d7f47928005d not found: ID does not exist" Nov 01 06:06:45.811398 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.811365 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r"] Nov 01 06:06:45.811816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.811802 4425 cpu_manager.go:414] "containerMap: removing container" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" containerName="test-curl-7443-5cc14f" Nov 01 06:06:45.811863 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.811817 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" containerName="test-curl-7443-5cc14f" Nov 01 06:06:45.811903 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.811890 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" containerName="test-curl-7443-5cc14f" Nov 01 06:06:45.816285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.816247 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.872566 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.872533 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r"] Nov 01 06:06:45.881274 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.881231 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqj4\" (UniqueName: \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.881430 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.881353 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.881430 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.881414 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.920549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.920514 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:45.920549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.920523 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:45.920764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.920590 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:45.920764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.920626 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:06:45.982523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.982492 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqj4\" (UniqueName: \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.982730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.982563 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.982730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.982617 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.983311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.983282 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:45.985160 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:45.985135 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:46.044559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:46.044528 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqj4\" (UniqueName: \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\") pod \"maas-api-revoke-sub-5edbba55-d2b4r\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:46.129219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:46.129136 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:46.505511 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:46.505474 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r"] Nov 01 06:06:46.507927 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:06:46.507892 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5098b415_11b6_412d_a565_c08bdde6b4ba.slice/crio-312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89 WatchSource:0}: Error finding container 312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89: Status 404 returned error can't find the container with id 312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89 Nov 01 06:06:47.246874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.246838 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" event={"ID":"5098b415-11b6-412d-a565-c08bdde6b4ba","Type":"ContainerStarted","Data":"35faa3492bba683d9e3f11c290eda723060148901bc533a5ea15d68f6d4ab375"} Nov 01 06:06:47.247356 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.246902 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" event={"ID":"5098b415-11b6-412d-a565-c08bdde6b4ba","Type":"ContainerStarted","Data":"312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89"} Nov 01 06:06:47.365896 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.365846 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" podStartSLOduration=2.3658325270000002 podStartE2EDuration="2.365832527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:47.364242469 +0000 UTC m=+3822.504831417" watchObservedRunningTime="2026-11-01 06:06:47.365832527 +0000 UTC m=+3822.506421443" Nov 01 06:06:47.458049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.458022 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:47.498376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.498294 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\" (UniqueName: \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\") pod \"042d825e-d6d1-48b8-b386-a3cbb8f6127c\" (UID: \"042d825e-d6d1-48b8-b386-a3cbb8f6127c\") " Nov 01 06:06:47.500488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.500465 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k" pod "042d825e-d6d1-48b8-b386-a3cbb8f6127c" (UID: "042d825e-d6d1-48b8-b386-a3cbb8f6127c"). InnerVolumeSpecName "kube-api-access-zbp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:47.599386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:47.599357 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbp2k\" (UniqueName: \"kubernetes.io/projected/042d825e-d6d1-48b8-b386-a3cbb8f6127c-kube-api-access-zbp2k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:48.253866 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.253769 4425 generic.go:356] "Generic (PLEG): container finished" podID="5098b415-11b6-412d-a565-c08bdde6b4ba" containerID="35faa3492bba683d9e3f11c290eda723060148901bc533a5ea15d68f6d4ab375" exitCode=0 Nov 01 06:06:48.254298 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.253857 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" event={"ID":"5098b415-11b6-412d-a565-c08bdde6b4ba","Type":"ContainerDied","Data":"35faa3492bba683d9e3f11c290eda723060148901bc533a5ea15d68f6d4ab375"} Nov 01 06:06:48.255224 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.255197 4425 generic.go:356] "Generic (PLEG): container finished" podID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" containerID="1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f" exitCode=137 Nov 01 06:06:48.255373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.255245 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:48.255373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.255289 4425 scope.go:122] "RemoveContainer" containerID="1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f" Nov 01 06:06:48.264459 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.264413 4425 scope.go:122] "RemoveContainer" containerID="1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f" Nov 01 06:06:48.264733 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:48.264715 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f\": container with ID starting with 1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f not found: ID does not exist" containerID="1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f" Nov 01 06:06:48.264785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.264742 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f"} err="failed to get container status \"1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f\": rpc error: code = NotFound desc = could not find container \"1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f\": container with ID starting with 1f49c06244f46e14711fb05d0e74870e82591c3c8733a947c2f454f780cc0b7f not found: ID does not exist" Nov 01 06:06:48.422872 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:48.422829 4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7443-5cc14f\" is forbidden: User \"system:node:ip-10-0-141-97.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-97.us-west-1.compute.internal' and this object" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" pod="opendatahub/test-curl-7443-5cc14f" Nov 01 06:06:48.972268 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:48.972227 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042d825e-d6d1-48b8-b386-a3cbb8f6127c" path="/var/lib/kubelet/pods/042d825e-d6d1-48b8-b386-a3cbb8f6127c/volumes" Nov 01 06:06:49.462275 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.453444 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:49.520730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.520628 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\") pod \"5098b415-11b6-412d-a565-c08bdde6b4ba\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " Nov 01 06:06:49.521110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.521067 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\" (UniqueName: \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\") pod \"5098b415-11b6-412d-a565-c08bdde6b4ba\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " Nov 01 06:06:49.521396 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.521377 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\") pod \"5098b415-11b6-412d-a565-c08bdde6b4ba\" (UID: \"5098b415-11b6-412d-a565-c08bdde6b4ba\") " Nov 01 06:06:49.522306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.522284 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca" pod "5098b415-11b6-412d-a565-c08bdde6b4ba" (UID: "5098b415-11b6-412d-a565-c08bdde6b4ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:49.538111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.538078 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4" pod "5098b415-11b6-412d-a565-c08bdde6b4ba" (UID: "5098b415-11b6-412d-a565-c08bdde6b4ba"). InnerVolumeSpecName "kube-api-access-fsqj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:49.539646 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.539601 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca" pod "5098b415-11b6-412d-a565-c08bdde6b4ba" (UID: "5098b415-11b6-412d-a565-c08bdde6b4ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:49.622647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.622608 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqj4\" (UniqueName: \"kubernetes.io/projected/5098b415-11b6-412d-a565-c08bdde6b4ba-kube-api-access-fsqj4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:49.622647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.622643 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:49.622647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:49.622654 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5098b415-11b6-412d-a565-c08bdde6b4ba-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:50.270880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:50.270836 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" event={"ID":"5098b415-11b6-412d-a565-c08bdde6b4ba","Type":"ContainerDied","Data":"312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89"} Nov 01 06:06:50.270880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:50.270873 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r" Nov 01 06:06:50.270880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:50.270891 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312fc02bf9982614a99e4efd9261dd9b18b08ece429b1b898a21f85d0ea02e89" Nov 01 06:06:56.618813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.618777 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc"] Nov 01 06:06:56.619293 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.619276 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5098b415-11b6-412d-a565-c08bdde6b4ba" containerName="revoke-keys" Nov 01 06:06:56.619355 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.619295 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5098b415-11b6-412d-a565-c08bdde6b4ba" containerName="revoke-keys" Nov 01 06:06:56.619400 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.619390 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5098b415-11b6-412d-a565-c08bdde6b4ba" containerName="revoke-keys" Nov 01 06:06:56.622821 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.622804 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.679238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.679206 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc"] Nov 01 06:06:56.693433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.693393 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6kx\" (UniqueName: \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.693630 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.693440 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.693630 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.693577 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.726902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.726864 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:06:56.726902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.726898 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:06:56.727108 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.726904 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:06:56.727108 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.726922 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:06:56.795266 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.795231 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6kx\" (UniqueName: \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.795452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.795289 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.795452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.795318 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.795957 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.795930 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.797929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.797906 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.857210 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.857181 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6kx\" (UniqueName: \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\") pod \"maas-api-revoke-sub-1d163946-btmxc\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.936078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:56.936003 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:56.947034 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:06:56.946992 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:06:57.311069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:57.311048 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc"] Nov 01 06:06:57.312908 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:06:57.312881 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e82711_b3ce_47c3_bca6_ac7bb754200e.slice/crio-d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec WatchSource:0}: Error finding container d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec: Status 404 returned error can't find the container with id d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec Nov 01 06:06:58.313664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:58.313636 4425 generic.go:356] "Generic (PLEG): container finished" podID="c9e82711-b3ce-47c3-bca6-ac7bb754200e" containerID="cbbe4be6e9b4aba7fb2f11c98ef3a682f3b3fe620b18e7d7074ad6fd61788860" exitCode=0 Nov 01 06:06:58.314127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:58.313726 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" event={"ID":"c9e82711-b3ce-47c3-bca6-ac7bb754200e","Type":"ContainerDied","Data":"cbbe4be6e9b4aba7fb2f11c98ef3a682f3b3fe620b18e7d7074ad6fd61788860"} Nov 01 06:06:58.314127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:58.313797 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" event={"ID":"c9e82711-b3ce-47c3-bca6-ac7bb754200e","Type":"ContainerStarted","Data":"d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec"} Nov 01 06:06:59.453881 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.453859 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:06:59.521871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.521837 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\") pod \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " Nov 01 06:06:59.522069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.521887 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\" (UniqueName: \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\") pod \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " Nov 01 06:06:59.522069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.522035 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\") pod \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\" (UID: \"c9e82711-b3ce-47c3-bca6-ac7bb754200e\") " Nov 01 06:06:59.522326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.522302 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca" pod "c9e82711-b3ce-47c3-bca6-ac7bb754200e" (UID: "c9e82711-b3ce-47c3-bca6-ac7bb754200e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:06:59.524159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.524133 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx" pod "c9e82711-b3ce-47c3-bca6-ac7bb754200e" (UID: "c9e82711-b3ce-47c3-bca6-ac7bb754200e"). InnerVolumeSpecName "kube-api-access-fj6kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:06:59.524301 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.524198 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca" pod "c9e82711-b3ce-47c3-bca6-ac7bb754200e" (UID: "c9e82711-b3ce-47c3-bca6-ac7bb754200e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:06:59.623576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.623497 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:59.623576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.623534 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6kx\" (UniqueName: \"kubernetes.io/projected/c9e82711-b3ce-47c3-bca6-ac7bb754200e-kube-api-access-fj6kx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:06:59.623576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:06:59.623551 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e82711-b3ce-47c3-bca6-ac7bb754200e-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:00.326997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:00.326966 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" Nov 01 06:07:00.327221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:00.327001 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc" event={"ID":"c9e82711-b3ce-47c3-bca6-ac7bb754200e","Type":"ContainerDied","Data":"d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec"} Nov 01 06:07:00.327221 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:00.327048 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35835b54d51dd8742d8a9a449e12948586791c241a808ace1f6fe593828bfec" Nov 01 06:07:03.034937 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:03.034898 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 06:07:03.035439 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:03.035105 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" containerID="cri-o://84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd" gracePeriod=30 Nov 01 06:07:03.514031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:03.513993 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-ztp4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:07:03.514224 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:03.514047 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:07:08.252550 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.252517 4425 scope.go:122] "RemoveContainer" containerID="d797e5c8b778377c7501f5f49e20129deaaba3f6c981721696ceedf4aca35f41" Nov 01 06:07:08.263945 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.263925 4425 scope.go:122] "RemoveContainer" containerID="aca2471b36ecfa021dc9cdfb7be7ae94566245cdfa109d69f4e0e0a0f8203640" Nov 01 06:07:08.300592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.300568 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 06:07:08.309193 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309132 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9"] Nov 01 06:07:08.309649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309630 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c9e82711-b3ce-47c3-bca6-ac7bb754200e" containerName="revoke-keys" Nov 01 06:07:08.309739 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309652 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e82711-b3ce-47c3-bca6-ac7bb754200e" containerName="revoke-keys" Nov 01 06:07:08.309739 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309669 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" Nov 01 06:07:08.309739 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309677 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" Nov 01 06:07:08.309914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309800 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e82711-b3ce-47c3-bca6-ac7bb754200e" containerName="revoke-keys" Nov 01 06:07:08.309914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.309816 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerName="istio-proxy" Nov 01 06:07:08.312949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.312931 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.367539 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.367490 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:08.367712 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.367555 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:08.372813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.372788 4425 generic.go:356] "Generic (PLEG): container finished" podID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" containerID="84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd" exitCode=0 Nov 01 06:07:08.372947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.372849 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" Nov 01 06:07:08.372947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.372862 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" event={"ID":"d2f3588d-5397-4aa5-b613-6b4fdf41bf52","Type":"ContainerDied","Data":"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd"} Nov 01 06:07:08.372947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.372922 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p" event={"ID":"d2f3588d-5397-4aa5-b613-6b4fdf41bf52","Type":"ContainerDied","Data":"9de5d2b1c7ffa9e366390a49a93b2087c0189933d1ab8ce5d0e815d357a02817"} Nov 01 06:07:08.373110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.372956 4425 scope.go:122] "RemoveContainer" containerID="84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd" Nov 01 06:07:08.383037 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.383019 4425 scope.go:122] "RemoveContainer" containerID="84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd" Nov 01 06:07:08.383381 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:07:08.383358 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd\": container with ID starting with 84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd not found: ID does not exist" containerID="84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd" Nov 01 06:07:08.383490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.383388 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd"} err="failed to get container status \"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd\": rpc error: code = NotFound desc = could not find container \"84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd\": container with ID starting with 84c3651f596b94ddddc9fc43a03fde71ff637f0c590bd792be78337340ceaadd not found: ID does not exist" Nov 01 06:07:08.404820 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.404793 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:08.405010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.404958 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:07:08.407511 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.407485 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9"] Nov 01 06:07:08.415951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.415921 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.415988 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416018 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416049 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416264 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416115 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416264 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416152 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416264 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416206 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416219 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:07:08.416450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416306 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416319 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:07:08.416450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416405 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\") pod \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\" (UID: \"d2f3588d-5397-4aa5-b613-6b4fdf41bf52\") " Nov 01 06:07:08.416683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416537 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:08.416683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416537 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:07:08.416683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416664 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.416854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416801 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:07:08.417153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.416964 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.417153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417008 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtkv\" (UniqueName: \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.417326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417126 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.417326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417245 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.417326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417310 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.417326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417323 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.417578 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.417339 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.418680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.418651 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:08.418680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.418666 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "kube-api-access-zxtrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:08.418813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.418674 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:07:08.418813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.418748 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo" pod "d2f3588d-5397-4aa5-b613-6b4fdf41bf52" (UID: "d2f3588d-5397-4aa5-b613-6b4fdf41bf52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:07:08.517907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.517869 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.518133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.517913 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtkv\" (UniqueName: \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.518133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.517981 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.518133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.518054 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxtrb\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-kube-api-access-zxtrb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.518133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.518071 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.518133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.518084 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.518374 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.518173 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f3588d-5397-4aa5-b613-6b4fdf41bf52-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:08.518545 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.518522 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.520541 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.520518 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.579971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.579944 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtkv\" (UniqueName: \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\") pod \"maas-api-revoke-sub-5954995f-xrjz9\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.625842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.625813 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:08.802010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.801975 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 06:07:08.805785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.805758 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-ztp4p"] Nov 01 06:07:08.972648 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:08.972567 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f3588d-5397-4aa5-b613-6b4fdf41bf52" path="/var/lib/kubelet/pods/d2f3588d-5397-4aa5-b613-6b4fdf41bf52/volumes" Nov 01 06:07:09.002350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:09.002326 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9"] Nov 01 06:07:09.003922 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:07:09.003891 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ecf889_64e8_48de_8868_7cb4b8785b3c.slice/crio-43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd WatchSource:0}: Error finding container 43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd: Status 404 returned error can't find the container with id 43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd Nov 01 06:07:09.380866 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:09.380834 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" event={"ID":"33ecf889-64e8-48de-8868-7cb4b8785b3c","Type":"ContainerStarted","Data":"52e619dc9e9987b6fbff01d9f379c539f5843be6beae8f86de1393a76cb60937"} Nov 01 06:07:09.381435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:09.380896 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" event={"ID":"33ecf889-64e8-48de-8868-7cb4b8785b3c","Type":"ContainerStarted","Data":"43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd"} Nov 01 06:07:09.499485 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:09.499362 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" podStartSLOduration=1.499345373 podStartE2EDuration="1.499345373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:09.497410276 +0000 UTC m=+3844.637999196" watchObservedRunningTime="2026-11-01 06:07:09.499345373 +0000 UTC m=+3844.639934292" Nov 01 06:07:10.387742 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:10.387704 4425 generic.go:356] "Generic (PLEG): container finished" podID="33ecf889-64e8-48de-8868-7cb4b8785b3c" containerID="52e619dc9e9987b6fbff01d9f379c539f5843be6beae8f86de1393a76cb60937" exitCode=0 Nov 01 06:07:10.388130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:10.387791 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" event={"ID":"33ecf889-64e8-48de-8868-7cb4b8785b3c","Type":"ContainerDied","Data":"52e619dc9e9987b6fbff01d9f379c539f5843be6beae8f86de1393a76cb60937"} Nov 01 06:07:11.548080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.548049 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:11.657009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.656921 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\" (UniqueName: \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\") pod \"33ecf889-64e8-48de-8868-7cb4b8785b3c\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " Nov 01 06:07:11.657009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.656962 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\") pod \"33ecf889-64e8-48de-8868-7cb4b8785b3c\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " Nov 01 06:07:11.657276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.657084 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\") pod \"33ecf889-64e8-48de-8868-7cb4b8785b3c\" (UID: \"33ecf889-64e8-48de-8868-7cb4b8785b3c\") " Nov 01 06:07:11.657500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.657476 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca" pod "33ecf889-64e8-48de-8868-7cb4b8785b3c" (UID: "33ecf889-64e8-48de-8868-7cb4b8785b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:11.659304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.659280 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca" pod "33ecf889-64e8-48de-8868-7cb4b8785b3c" (UID: "33ecf889-64e8-48de-8868-7cb4b8785b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:11.659402 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.659314 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv" pod "33ecf889-64e8-48de-8868-7cb4b8785b3c" (UID: "33ecf889-64e8-48de-8868-7cb4b8785b3c"). InnerVolumeSpecName "kube-api-access-5jtkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:11.758337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.758292 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:11.758337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.758327 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtkv\" (UniqueName: \"kubernetes.io/projected/33ecf889-64e8-48de-8868-7cb4b8785b3c-kube-api-access-5jtkv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:11.758337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:11.758341 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ecf889-64e8-48de-8868-7cb4b8785b3c-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:12.400933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:12.400874 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" Nov 01 06:07:12.401147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:12.400873 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9" event={"ID":"33ecf889-64e8-48de-8868-7cb4b8785b3c","Type":"ContainerDied","Data":"43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd"} Nov 01 06:07:12.401147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:12.401015 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43daa9855fde5ac0f41eb53e53665bfac88dca83216a4561e59104485d5934dd" Nov 01 06:07:19.208718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.208683 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87"] Nov 01 06:07:19.209290 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.209108 4425 cpu_manager.go:414] "containerMap: removing container" podUID="33ecf889-64e8-48de-8868-7cb4b8785b3c" containerName="revoke-keys" Nov 01 06:07:19.209290 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.209119 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ecf889-64e8-48de-8868-7cb4b8785b3c" containerName="revoke-keys" Nov 01 06:07:19.209290 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.209221 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ecf889-64e8-48de-8868-7cb4b8785b3c" containerName="revoke-keys" Nov 01 06:07:19.212375 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.212358 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.224521 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.224499 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7g6n\" (UniqueName: \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.224641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.224554 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.224779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.224757 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.269152 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.269122 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87"] Nov 01 06:07:19.317166 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.317126 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:07:19.317166 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.317144 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:07:19.317418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.317179 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:07:19.317418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.317199 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\"" Nov 01 06:07:19.325698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.325675 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.325837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.325723 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7g6n\" (UniqueName: \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.325837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.325761 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.326480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.326458 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.328235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.328215 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.387748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.387719 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7g6n\" (UniqueName: \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\") pod \"maas-api-revoke-sub-1a0ce180-88n87\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.525833 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.525754 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:19.900540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:19.900510 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87"] Nov 01 06:07:19.901777 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:07:19.901748 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b10d240_ed5e_450c_b8cc_f59ac95db8e4.slice/crio-8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d WatchSource:0}: Error finding container 8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d: Status 404 returned error can't find the container with id 8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d Nov 01 06:07:20.445688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:20.445592 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" event={"ID":"2b10d240-ed5e-450c-b8cc-f59ac95db8e4","Type":"ContainerStarted","Data":"6f4077c6d2f022b25a4e93f2a077b9b8ed383437804c9bc42bcfd5d309b18c30"} Nov 01 06:07:20.445688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:20.445658 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" event={"ID":"2b10d240-ed5e-450c-b8cc-f59ac95db8e4","Type":"ContainerStarted","Data":"8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d"} Nov 01 06:07:20.562893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:20.562840 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" podStartSLOduration=1.562826454 podStartE2EDuration="1.562826454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:20.561945044 +0000 UTC m=+3855.702533962" watchObservedRunningTime="2026-11-01 06:07:20.562826454 +0000 UTC m=+3855.703415406" Nov 01 06:07:21.452940 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:21.452904 4425 generic.go:356] "Generic (PLEG): container finished" podID="2b10d240-ed5e-450c-b8cc-f59ac95db8e4" containerID="6f4077c6d2f022b25a4e93f2a077b9b8ed383437804c9bc42bcfd5d309b18c30" exitCode=0 Nov 01 06:07:21.453387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:21.452989 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" event={"ID":"2b10d240-ed5e-450c-b8cc-f59ac95db8e4","Type":"ContainerDied","Data":"6f4077c6d2f022b25a4e93f2a077b9b8ed383437804c9bc42bcfd5d309b18c30"} Nov 01 06:07:22.604062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.604037 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:22.657806 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.657770 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\") pod \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " Nov 01 06:07:22.657990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.657870 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\" (UniqueName: \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\") pod \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " Nov 01 06:07:22.658043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.657993 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\") pod \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\" (UID: \"2b10d240-ed5e-450c-b8cc-f59ac95db8e4\") " Nov 01 06:07:22.658222 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.658192 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca" pod "2b10d240-ed5e-450c-b8cc-f59ac95db8e4" (UID: "2b10d240-ed5e-450c-b8cc-f59ac95db8e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:07:22.658431 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.658414 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:22.660201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.660174 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n" pod "2b10d240-ed5e-450c-b8cc-f59ac95db8e4" (UID: "2b10d240-ed5e-450c-b8cc-f59ac95db8e4"). InnerVolumeSpecName "kube-api-access-v7g6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:07:22.660329 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.660294 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca" pod "2b10d240-ed5e-450c-b8cc-f59ac95db8e4" (UID: "2b10d240-ed5e-450c-b8cc-f59ac95db8e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:07:22.759834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.759749 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7g6n\" (UniqueName: \"kubernetes.io/projected/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-kube-api-access-v7g6n\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:22.759834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:22.759778 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b10d240-ed5e-450c-b8cc-f59ac95db8e4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:07:23.465444 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:23.465401 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" event={"ID":"2b10d240-ed5e-450c-b8cc-f59ac95db8e4","Type":"ContainerDied","Data":"8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d"} Nov 01 06:07:23.465638 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:23.465469 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d86e1d721e15955d0cf06efc8dde0ba0e57785d1b4153ab535c51dae1a9e78d" Nov 01 06:07:23.465638 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:23.465427 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87" Nov 01 06:07:26.947079 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:07:26.947039 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:07:49.721394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:49.721363 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:07:55.577356 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:55.577316 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 06:07:55.687113 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:07:55.687085 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 06:07:56.947657 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:07:56.947615 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:08:00.577755 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:00.577704 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" containerID="cri-o://2745b5548e2d93c77a4322c3c6570d82da7d1919a774b9e6abc78178a0744db6" gracePeriod=25 Nov 01 06:08:00.687411 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:00.687372 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" containerID="cri-o://e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8" gracePeriod=25 Nov 01 06:08:04.741509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:04.741463 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" podUID="70255189-5f34-4c75-9689-b523fecb5bef" containerName="main" containerID="cri-o://9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff" gracePeriod=45 Nov 01 06:08:05.001332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.001272 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:08:05.076924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.076889 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\" (UniqueName: \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.076985 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077015 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077038 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077123 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077166 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\" (UniqueName: \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\") pod \"70255189-5f34-4c75-9689-b523fecb5bef\" (UID: \"70255189-5f34-4c75-9689-b523fecb5bef\") " Nov 01 06:08:05.077308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077286 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:08:05.077500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077393 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:08:05.077500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077411 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:08:05.077616 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077546 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.077616 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077578 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.077616 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.077593 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.079575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.079553 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "kube-api-access-pmfh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:05.079702 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.079686 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:08:05.079752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.079708 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm" pod "70255189-5f34-4c75-9689-b523fecb5bef" (UID: "70255189-5f34-4c75-9689-b523fecb5bef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:08:05.178326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.178292 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70255189-5f34-4c75-9689-b523fecb5bef-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.178326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.178320 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70255189-5f34-4c75-9689-b523fecb5bef-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.178326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.178330 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfh6\" (UniqueName: \"kubernetes.io/projected/70255189-5f34-4c75-9689-b523fecb5bef-kube-api-access-pmfh6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:05.708836 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.708802 4425 generic.go:356] "Generic (PLEG): container finished" podID="70255189-5f34-4c75-9689-b523fecb5bef" containerID="9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff" exitCode=0 Nov 01 06:08:05.709038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.708870 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" Nov 01 06:08:05.709038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.708885 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" event={"ID":"70255189-5f34-4c75-9689-b523fecb5bef","Type":"ContainerDied","Data":"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff"} Nov 01 06:08:05.709038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.708939 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc" event={"ID":"70255189-5f34-4c75-9689-b523fecb5bef","Type":"ContainerDied","Data":"95e4673119ec629e157fcc212ad9319507307097b0127a27efe724589dc58236"} Nov 01 06:08:05.709038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.708962 4425 scope.go:122] "RemoveContainer" containerID="9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff" Nov 01 06:08:05.718923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.718899 4425 scope.go:122] "RemoveContainer" containerID="9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff" Nov 01 06:08:05.719196 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:08:05.719176 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff\": container with ID starting with 9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff not found: ID does not exist" containerID="9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff" Nov 01 06:08:05.719246 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.719205 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff"} err="failed to get container status \"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff\": rpc error: code = NotFound desc = could not find container \"9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff\": container with ID starting with 9a1d253c2ebe811bf42c8380f6b37f6c43b7d11872adeef9df1478ce5fc77fff not found: ID does not exist" Nov 01 06:08:05.877360 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.877326 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:08:05.927957 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:05.927928 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fd2672/sub-test-model-fd2672-kserve-7bf65989d7-6nlsc"] Nov 01 06:08:06.972654 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:06.972621 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70255189-5f34-4c75-9689-b523fecb5bef" path="/var/lib/kubelet/pods/70255189-5f34-4c75-9689-b523fecb5bef/volumes" Nov 01 06:08:08.300426 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:08.300399 4425 scope.go:122] "RemoveContainer" containerID="dbe68f1bef48428375be7cc2ad5fa62327755cf491ce216fc6668a5e151a06d1" Nov 01 06:08:08.311932 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:08.311913 4425 scope.go:122] "RemoveContainer" containerID="6d0e1504366144feb866607fdedac733de69254f63966c31df22c22bc6d086dd" Nov 01 06:08:08.516929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:08.516890 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:09.522940 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:09.522895 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:18.516297 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:18.516192 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:19.523348 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:19.523309 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:20.049272 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.049229 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw"] Nov 01 06:08:20.099862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.099828 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6378ec37-lbdgw"] Nov 01 06:08:20.796683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.796623 4425 generic.go:356] "Generic (PLEG): container finished" podID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerID="2745b5548e2d93c77a4322c3c6570d82da7d1919a774b9e6abc78178a0744db6" exitCode=0 Nov 01 06:08:20.797445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.796794 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" event={"ID":"dd96a9f4-051c-4089-aa38-85233c9b7715","Type":"ContainerDied","Data":"2745b5548e2d93c77a4322c3c6570d82da7d1919a774b9e6abc78178a0744db6"} Nov 01 06:08:20.841693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.841626 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 06:08:20.935308 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.935280 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\" (UniqueName: \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\") pod \"dd96a9f4-051c-4089-aa38-85233c9b7715\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " Nov 01 06:08:20.935450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.935360 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\") pod \"dd96a9f4-051c-4089-aa38-85233c9b7715\" (UID: \"dd96a9f4-051c-4089-aa38-85233c9b7715\") " Nov 01 06:08:20.935776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.935750 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume" pod "dd96a9f4-051c-4089-aa38-85233c9b7715" (UID: "dd96a9f4-051c-4089-aa38-85233c9b7715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:20.936289 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.936272 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 06:08:20.937435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.937414 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g" pod "dd96a9f4-051c-4089-aa38-85233c9b7715" (UID: "dd96a9f4-051c-4089-aa38-85233c9b7715"). InnerVolumeSpecName "kube-api-access-ztr4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:20.973430 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:20.973401 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aadf8b3-0542-44f2-87e8-80c7dccf1530" path="/var/lib/kubelet/pods/3aadf8b3-0542-44f2-87e8-80c7dccf1530/volumes" Nov 01 06:08:21.035859 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.035831 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\" (UniqueName: \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\") pod \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " Nov 01 06:08:21.035859 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.035875 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\") pod \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\" (UID: \"e8b0971a-4f88-4acc-aa4b-51f084de7c70\") " Nov 01 06:08:21.036218 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.036189 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztr4g\" (UniqueName: \"kubernetes.io/projected/dd96a9f4-051c-4089-aa38-85233c9b7715-kube-api-access-ztr4g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:21.036218 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.036209 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd96a9f4-051c-4089-aa38-85233c9b7715-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:21.036417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.036293 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume" pod "e8b0971a-4f88-4acc-aa4b-51f084de7c70" (UID: "e8b0971a-4f88-4acc-aa4b-51f084de7c70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:21.038359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.038336 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8" pod "e8b0971a-4f88-4acc-aa4b-51f084de7c70" (UID: "e8b0971a-4f88-4acc-aa4b-51f084de7c70"). InnerVolumeSpecName "kube-api-access-mptt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:21.137206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.137134 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptt8\" (UniqueName: \"kubernetes.io/projected/e8b0971a-4f88-4acc-aa4b-51f084de7c70-kube-api-access-mptt8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:21.137206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.137160 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b0971a-4f88-4acc-aa4b-51f084de7c70-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:21.803816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.803782 4425 generic.go:356] "Generic (PLEG): container finished" podID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerID="e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8" exitCode=0 Nov 01 06:08:21.804309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.803829 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" Nov 01 06:08:21.804309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.803878 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" event={"ID":"e8b0971a-4f88-4acc-aa4b-51f084de7c70","Type":"ContainerDied","Data":"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8"} Nov 01 06:08:21.804309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.803931 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5" event={"ID":"e8b0971a-4f88-4acc-aa4b-51f084de7c70","Type":"ContainerDied","Data":"331923b7d810b528e7d12c87b110f2cad2681c8a1a54a1a3a0dcf2a71e72e43e"} Nov 01 06:08:21.804309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.803948 4425 scope.go:122] "RemoveContainer" containerID="e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8" Nov 01 06:08:21.805588 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.805566 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" event={"ID":"dd96a9f4-051c-4089-aa38-85233c9b7715","Type":"ContainerDied","Data":"a7dfc741efb97c91382cfdaf2cefc3c61b43e49f442957bf40113cb0a76baedd"} Nov 01 06:08:21.805588 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.805582 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7" Nov 01 06:08:21.815062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.815041 4425 scope.go:122] "RemoveContainer" containerID="e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8" Nov 01 06:08:21.815370 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:08:21.815351 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8\": container with ID starting with e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8 not found: ID does not exist" containerID="e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8" Nov 01 06:08:21.815432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.815379 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8"} err="failed to get container status \"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8\": rpc error: code = NotFound desc = could not find container \"e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8\": container with ID starting with e720c5bbd623510f8e2b70f193932d3d76961c1ad4ac936329ef2e83bd0545f8 not found: ID does not exist" Nov 01 06:08:21.815432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.815397 4425 scope.go:122] "RemoveContainer" containerID="2745b5548e2d93c77a4322c3c6570d82da7d1919a774b9e6abc78178a0744db6" Nov 01 06:08:21.971054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:21.970959 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 06:08:22.021638 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:22.021605 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-897c87-57666cd899-bjrv7"] Nov 01 06:08:22.133312 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:22.133276 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 06:08:22.139111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:22.139088 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-897c87-96bc99f-jr2x5"] Nov 01 06:08:22.972498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:22.972467 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" path="/var/lib/kubelet/pods/dd96a9f4-051c-4089-aa38-85233c9b7715/volumes" Nov 01 06:08:22.973027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:22.973010 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" path="/var/lib/kubelet/pods/e8b0971a-4f88-4acc-aa4b-51f084de7c70/volumes" Nov 01 06:08:26.947755 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:08:26.947709 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:08:31.793117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:31.793072 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 06:08:31.903283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:31.903220 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 06:08:36.058042 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:36.058007 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt"] Nov 01 06:08:36.109058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:36.109020 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff5b6f0d-ftkbt"] Nov 01 06:08:36.793602 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:36.793559 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" containerID="cri-o://74697569666c491e4fa7de7f50dfb3234a61a8661df0302f69ce857a763fc04f" gracePeriod=25 Nov 01 06:08:36.904294 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:36.904231 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" containerID="cri-o://cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99" gracePeriod=25 Nov 01 06:08:36.972919 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:36.972881 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95198ef-c5d3-42a1-8716-92fbfe2865fb" path="/var/lib/kubelet/pods/f95198ef-c5d3-42a1-8716-92fbfe2865fb/volumes" Nov 01 06:08:41.114684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:41.114641 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:41.115149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:41.114645 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:51.114530 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:51.114493 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:51.114530 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:51.114518 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:08:53.450128 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:53.450091 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 06:08:56.952668 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:08:56.952630 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:08:56.998542 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:56.998509 4425 generic.go:356] "Generic (PLEG): container finished" podID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerID="74697569666c491e4fa7de7f50dfb3234a61a8661df0302f69ce857a763fc04f" exitCode=0 Nov 01 06:08:56.998708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:56.998586 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" event={"ID":"90e904aa-5f45-4e1c-ab54-c2c8e6170958","Type":"ContainerDied","Data":"74697569666c491e4fa7de7f50dfb3234a61a8661df0302f69ce857a763fc04f"} Nov 01 06:08:57.058790 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.058723 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 06:08:57.170127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.170097 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 06:08:57.195418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.195381 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\" (UniqueName: \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\") pod \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " Nov 01 06:08:57.195642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.195523 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\") pod \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\" (UID: \"90e904aa-5f45-4e1c-ab54-c2c8e6170958\") " Nov 01 06:08:57.196115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.196082 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume" pod "90e904aa-5f45-4e1c-ab54-c2c8e6170958" (UID: "90e904aa-5f45-4e1c-ab54-c2c8e6170958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:57.198160 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.198136 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp" pod "90e904aa-5f45-4e1c-ab54-c2c8e6170958" (UID: "90e904aa-5f45-4e1c-ab54-c2c8e6170958"). InnerVolumeSpecName "kube-api-access-9mkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:57.297042 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.297011 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\" (UniqueName: \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\") pod \"040645ce-dc42-4069-b610-9554f4444e52\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " Nov 01 06:08:57.297285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.297090 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\") pod \"040645ce-dc42-4069-b610-9554f4444e52\" (UID: \"040645ce-dc42-4069-b610-9554f4444e52\") " Nov 01 06:08:57.297454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.297427 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkhp\" (UniqueName: \"kubernetes.io/projected/90e904aa-5f45-4e1c-ab54-c2c8e6170958-kube-api-access-9mkhp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:57.297454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.297456 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e904aa-5f45-4e1c-ab54-c2c8e6170958-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:57.297677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.297606 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume" pod "040645ce-dc42-4069-b610-9554f4444e52" (UID: "040645ce-dc42-4069-b610-9554f4444e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:08:57.299810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.299786 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl" pod "040645ce-dc42-4069-b610-9554f4444e52" (UID: "040645ce-dc42-4069-b610-9554f4444e52"). InnerVolumeSpecName "kube-api-access-rnndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:08:57.398449 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.398414 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnndl\" (UniqueName: \"kubernetes.io/projected/040645ce-dc42-4069-b610-9554f4444e52-kube-api-access-rnndl\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:57.398449 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:57.398444 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040645ce-dc42-4069-b610-9554f4444e52-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:08:58.007768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.007726 4425 generic.go:356] "Generic (PLEG): container finished" podID="040645ce-dc42-4069-b610-9554f4444e52" containerID="cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99" exitCode=0 Nov 01 06:08:58.008391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.007785 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" Nov 01 06:08:58.008391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.007814 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" event={"ID":"040645ce-dc42-4069-b610-9554f4444e52","Type":"ContainerDied","Data":"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99"} Nov 01 06:08:58.008391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.007879 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d" event={"ID":"040645ce-dc42-4069-b610-9554f4444e52","Type":"ContainerDied","Data":"154c77da1250985fc93df1ffb1ef7bc9d8fd435cb463756c9b10b42a7697d801"} Nov 01 06:08:58.008391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.007902 4425 scope.go:122] "RemoveContainer" containerID="cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99" Nov 01 06:08:58.009894 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.009870 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" event={"ID":"90e904aa-5f45-4e1c-ab54-c2c8e6170958","Type":"ContainerDied","Data":"00b32976e4b6ea1a382ad073d971877b20b410d5990eff9e78314ddb2fc78450"} Nov 01 06:08:58.009894 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.009876 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2" Nov 01 06:08:58.022560 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.019795 4425 scope.go:122] "RemoveContainer" containerID="cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99" Nov 01 06:08:58.022788 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:08:58.022715 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99\": container with ID starting with cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99 not found: ID does not exist" containerID="cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99" Nov 01 06:08:58.022788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.022746 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99"} err="failed to get container status \"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99\": rpc error: code = NotFound desc = could not find container \"cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99\": container with ID starting with cf569696b7e6c36c73045f2e6f583ce57d66c38300b492284f66ce93bd2a9a99 not found: ID does not exist" Nov 01 06:08:58.022788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.022771 4425 scope.go:122] "RemoveContainer" containerID="74697569666c491e4fa7de7f50dfb3234a61a8661df0302f69ce857a763fc04f" Nov 01 06:08:58.179118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.179012 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 06:08:58.185181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.185145 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a12c35-6dc5c85cff-wbz7d"] Nov 01 06:08:58.295715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.295672 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 06:08:58.300089 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.300058 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a12c35-6cdb45475c-brsf2"] Nov 01 06:08:58.451385 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.451291 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" containerID="cri-o://9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3" gracePeriod=25 Nov 01 06:08:58.972379 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.972350 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040645ce-dc42-4069-b610-9554f4444e52" path="/var/lib/kubelet/pods/040645ce-dc42-4069-b610-9554f4444e52/volumes" Nov 01 06:08:58.972744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:58.972729 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" path="/var/lib/kubelet/pods/90e904aa-5f45-4e1c-ab54-c2c8e6170958/volumes" Nov 01 06:08:59.691932 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:08:59.691894 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:09:08.373229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:08.373197 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-847cf6dc9d-lk8pc_085b04bc-51a6-4928-815d-f38fe27aea3d/authorino/0.log" Nov 01 06:09:09.692094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:09.692052 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:09:12.200732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:12.200703 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/0.log" Nov 01 06:09:18.700093 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.700057 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 06:09:18.795542 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.795457 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\" (UniqueName: \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\") pod \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " Nov 01 06:09:18.795542 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.795492 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\") pod \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\" (UID: \"5cfa7b5c-1fac-456b-b04d-702ff8a050c4\") " Nov 01 06:09:18.795916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.795892 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume" pod "5cfa7b5c-1fac-456b-b04d-702ff8a050c4" (UID: "5cfa7b5c-1fac-456b-b04d-702ff8a050c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:09:18.797872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.797847 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7" pod "5cfa7b5c-1fac-456b-b04d-702ff8a050c4" (UID: "5cfa7b5c-1fac-456b-b04d-702ff8a050c4"). InnerVolumeSpecName "kube-api-access-hwtn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:09:18.896467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.896433 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwtn7\" (UniqueName: \"kubernetes.io/projected/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-kube-api-access-hwtn7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:09:18.896467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:18.896460 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cfa7b5c-1fac-456b-b04d-702ff8a050c4-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:09:19.122999 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.122964 4425 generic.go:356] "Generic (PLEG): container finished" podID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerID="9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3" exitCode=0 Nov 01 06:09:19.123211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.123009 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" Nov 01 06:09:19.123211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.123013 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" event={"ID":"5cfa7b5c-1fac-456b-b04d-702ff8a050c4","Type":"ContainerDied","Data":"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3"} Nov 01 06:09:19.123211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.123067 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r" event={"ID":"5cfa7b5c-1fac-456b-b04d-702ff8a050c4","Type":"ContainerDied","Data":"8b44f4d3d441b986b7e1f7331d16ce75c7983447ca241e558578430e415a49d1"} Nov 01 06:09:19.123211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.123095 4425 scope.go:122] "RemoveContainer" containerID="9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3" Nov 01 06:09:19.132178 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.132158 4425 scope.go:122] "RemoveContainer" containerID="9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3" Nov 01 06:09:19.132469 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:09:19.132451 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3\": container with ID starting with 9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3 not found: ID does not exist" containerID="9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3" Nov 01 06:09:19.132527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.132476 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3"} err="failed to get container status \"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3\": rpc error: code = NotFound desc = could not find container \"9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3\": container with ID starting with 9aa7b785a16af5dfbaafd778ec34d0c8e5e4b9d17a72c8f7b18d7f068c91e7c3 not found: ID does not exist" Nov 01 06:09:19.292767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.292730 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 06:09:19.298906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:19.298878 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2fd4c9-5d6bc9cdff-j8c2r"] Nov 01 06:09:20.972502 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:20.972460 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" path="/var/lib/kubelet/pods/5cfa7b5c-1fac-456b-b04d-702ff8a050c4/volumes" Nov 01 06:09:26.949159 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:09:26.949120 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:09:44.046552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:44.046507 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw"] Nov 01 06:09:44.049832 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:44.049805 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36acd0e6-2hgdw"] Nov 01 06:09:44.972585 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:44.972550 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9577c589-e5d2-4bec-a4b0-eb7c18ae76cf" path="/var/lib/kubelet/pods/9577c589-e5d2-4bec-a4b0-eb7c18ae76cf/volumes" Nov 01 06:09:56.047402 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:56.047316 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv"] Nov 01 06:09:56.098031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:56.097996 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2803f-lc2bv"] Nov 01 06:09:56.946661 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:09:56.946628 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:09:56.971715 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:56.971686 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb189d7e-39c8-4da5-8719-05d60bddbf3c" path="/var/lib/kubelet/pods/cb189d7e-39c8-4da5-8719-05d60bddbf3c/volumes" Nov 01 06:09:59.555596 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555559 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn"] Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555946 4425 cpu_manager.go:414] "containerMap: removing container" podUID="70255189-5f34-4c75-9689-b523fecb5bef" containerName="main" Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555956 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70255189-5f34-4c75-9689-b523fecb5bef" containerName="main" Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555975 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555980 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.555997 4425 cpu_manager.go:414] "containerMap: removing container" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" Nov 01 06:09:59.556001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556002 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556014 4425 cpu_manager.go:414] "containerMap: removing container" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556019 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556033 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556039 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556045 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2b10d240-ed5e-450c-b8cc-f59ac95db8e4" containerName="revoke-keys" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556050 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b10d240-ed5e-450c-b8cc-f59ac95db8e4" containerName="revoke-keys" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556060 4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556065 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556121 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd96a9f4-051c-4089-aa38-85233c9b7715" containerName="payload-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556131 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="70255189-5f34-4c75-9689-b523fecb5bef" containerName="main" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556138 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b0971a-4f88-4acc-aa4b-51f084de7c70" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556145 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e904aa-5f45-4e1c-ab54-c2c8e6170958" containerName="payload-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556151 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="040645ce-dc42-4069-b610-9554f4444e52" containerName="payload-pre-processing" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556158 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b10d240-ed5e-450c-b8cc-f59ac95db8e4" containerName="revoke-keys" Nov 01 06:09:59.556212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.556163 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfa7b5c-1fac-456b-b04d-702ff8a050c4" containerName="payload-pre-processing" Nov 01 06:09:59.559512 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.559495 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.587506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.587469 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7p4\" (UniqueName: \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.587683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.587544 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.587683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.587665 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.613571 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.613535 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:09:59.616183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.616159 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn"] Nov 01 06:09:59.664517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.664484 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:09:59.664517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.664515 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:09:59.688876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.688827 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.689073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.688930 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7p4\" (UniqueName: \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.689073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.688983 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.689712 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.689692 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.691415 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.691399 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.714046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.714017 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:09:59.751060 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.751023 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7p4\" (UniqueName: \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\") pod \"maas-api-revoke-keys-e2e-ait-3951eb77-mxntn\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:09:59.872785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:09:59.872750 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:10:00.246445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:00.246418 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn"] Nov 01 06:10:00.247944 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:10:00.247907 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae32a6c5_70c1_421b_90de_b9df11355038.slice/crio-e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d WatchSource:0}: Error finding container e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d: Status 404 returned error can't find the container with id e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d Nov 01 06:10:00.334094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:00.334057 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" event={"ID":"ae32a6c5-70c1-421b-90de-b9df11355038","Type":"ContainerStarted","Data":"e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d"} Nov 01 06:10:01.341201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:01.341163 4425 generic.go:356] "Generic (PLEG): container finished" podID="ae32a6c5-70c1-421b-90de-b9df11355038" containerID="9b21fa052a7f15f19795eb09528aab68fda4523711837a54d98ac3e7e808958a" exitCode=0 Nov 01 06:10:01.341704 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:01.341263 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" event={"ID":"ae32a6c5-70c1-421b-90de-b9df11355038","Type":"ContainerDied","Data":"9b21fa052a7f15f19795eb09528aab68fda4523711837a54d98ac3e7e808958a"} Nov 01 06:10:02.487733 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.487703 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:10:02.514961 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.514937 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\") pod \"ae32a6c5-70c1-421b-90de-b9df11355038\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " Nov 01 06:10:02.515119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.515009 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\") pod \"ae32a6c5-70c1-421b-90de-b9df11355038\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " Nov 01 06:10:02.515119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.515039 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\" (UniqueName: \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\") pod \"ae32a6c5-70c1-421b-90de-b9df11355038\" (UID: \"ae32a6c5-70c1-421b-90de-b9df11355038\") " Nov 01 06:10:02.515414 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.515384 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca" pod "ae32a6c5-70c1-421b-90de-b9df11355038" (UID: "ae32a6c5-70c1-421b-90de-b9df11355038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:02.517287 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.517245 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca" pod "ae32a6c5-70c1-421b-90de-b9df11355038" (UID: "ae32a6c5-70c1-421b-90de-b9df11355038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:02.517391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.517364 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4" pod "ae32a6c5-70c1-421b-90de-b9df11355038" (UID: "ae32a6c5-70c1-421b-90de-b9df11355038"). InnerVolumeSpecName "kube-api-access-lx7p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:02.616717 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.616648 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:02.616717 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.616672 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7p4\" (UniqueName: \"kubernetes.io/projected/ae32a6c5-70c1-421b-90de-b9df11355038-kube-api-access-lx7p4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:02.616717 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:02.616689 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae32a6c5-70c1-421b-90de-b9df11355038-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:03.353228 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:03.353197 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" Nov 01 06:10:03.353228 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:03.353212 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn" event={"ID":"ae32a6c5-70c1-421b-90de-b9df11355038","Type":"ContainerDied","Data":"e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d"} Nov 01 06:10:03.353466 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:03.353281 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e7037b1fc8c622fa5113a052f14985b960c543d6aba81705b16c11ef403c5d" Nov 01 06:10:06.046222 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:06.046170 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242"] Nov 01 06:10:06.097045 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:06.097014 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9d67e1-hw242"] Nov 01 06:10:06.971768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:06.971734 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6528f8-017a-4945-ba0a-ce00c7469a73" path="/var/lib/kubelet/pods/dc6528f8-017a-4945-ba0a-ce00c7469a73/volumes" Nov 01 06:10:08.373617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:08.373583 4425 scope.go:122] "RemoveContainer" containerID="08aaf1a810881281ef9941588766319b32431946b2f7aa9d6fb9ec6dcf8666ab" Nov 01 06:10:08.383032 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:08.383007 4425 scope.go:122] "RemoveContainer" containerID="acd5cbe3de6d8c2d4ed60d35592daaba2634209ca4f28d923ac536f4f132495e" Nov 01 06:10:16.046665 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:16.046631 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7"] Nov 01 06:10:16.097215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:16.097185 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d67f334e-c68n7"] Nov 01 06:10:16.973127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:16.973091 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729ff929-02f5-42b3-82f9-3afaca30f5b8" path="/var/lib/kubelet/pods/729ff929-02f5-42b3-82f9-3afaca30f5b8/volumes" Nov 01 06:10:19.273111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:19.273075 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn"] Nov 01 06:10:19.323809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:19.323776 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3951eb77-mxntn"] Nov 01 06:10:20.972343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:20.972300 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae32a6c5-70c1-421b-90de-b9df11355038" path="/var/lib/kubelet/pods/ae32a6c5-70c1-421b-90de-b9df11355038/volumes" Nov 01 06:10:26.947835 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:10:26.947790 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:10:29.047008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:29.046974 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn"] Nov 01 06:10:29.097768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:29.097738 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e16b75f1-zs4kn"] Nov 01 06:10:30.975392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:30.975348 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fedf85-f5bf-4d05-8e0b-765999948026" path="/var/lib/kubelet/pods/13fedf85-f5bf-4d05-8e0b-765999948026/volumes" Nov 01 06:10:34.639489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:34.639447 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 06:10:34.640015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:34.639714 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" containerID="cri-o://068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876" gracePeriod=30 Nov 01 06:10:39.047173 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.047139 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp"] Nov 01 06:10:39.097855 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.097822 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f638148b-v6gjp"] Nov 01 06:10:39.897330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.897305 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 06:10:39.975417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975329 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975369 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975389 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975409 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975472 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975525 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975606 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975634 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:39.975762 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975740 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:39.976048 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975772 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:39.976048 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975858 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:39.976048 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.975913 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:39.976048 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.976032 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:39.976226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.976054 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8787e714-bc21-4665-b904-6c521c49244e-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:39.976226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.976076 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:39.976226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.976093 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:39.976226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.976107 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:39.977901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.977877 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:10:39.977986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.977881 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:39.978068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:39.978051 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "kube-api-access-6shk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:40.076471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.076426 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\") pod \"8787e714-bc21-4665-b904-6c521c49244e\" (UID: \"8787e714-bc21-4665-b904-6c521c49244e\") " Nov 01 06:10:40.076997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.076796 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:40.076997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.076819 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shk9\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-kube-api-access-6shk9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:40.076997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.076835 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8787e714-bc21-4665-b904-6c521c49244e-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:40.076997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.076853 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8787e714-bc21-4665-b904-6c521c49244e-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:40.078925 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.078896 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy" pod "8787e714-bc21-4665-b904-6c521c49244e" (UID: "8787e714-bc21-4665-b904-6c521c49244e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:40.177370 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.177335 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8787e714-bc21-4665-b904-6c521c49244e-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:40.543624 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.543586 4425 generic.go:356] "Generic (PLEG): container finished" podID="8787e714-bc21-4665-b904-6c521c49244e" containerID="068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876" exitCode=0 Nov 01 06:10:40.543624 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.543623 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" event={"ID":"8787e714-bc21-4665-b904-6c521c49244e","Type":"ContainerDied","Data":"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876"} Nov 01 06:10:40.543887 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.543642 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" Nov 01 06:10:40.543887 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.543663 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb" event={"ID":"8787e714-bc21-4665-b904-6c521c49244e","Type":"ContainerDied","Data":"9b4358e81716cdcfe59d4db48659e0bdb262fb1f25706121939736cb962debe2"} Nov 01 06:10:40.543887 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.543684 4425 scope.go:122] "RemoveContainer" containerID="068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876" Nov 01 06:10:40.553329 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.553307 4425 scope.go:122] "RemoveContainer" containerID="068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876" Nov 01 06:10:40.553614 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:10:40.553596 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876\": container with ID starting with 068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876 not found: ID does not exist" containerID="068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876" Nov 01 06:10:40.553666 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.553627 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876"} err="failed to get container status \"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876\": rpc error: code = NotFound desc = could not find container \"068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876\": container with ID starting with 068f6cb01a11e2404bd40e9b4574274f817e61c7b86291b890167e447e96a876 not found: ID does not exist" Nov 01 06:10:40.710734 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.710692 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 06:10:40.761694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.761658 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-sd9zb"] Nov 01 06:10:40.972327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.972291 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8787e714-bc21-4665-b904-6c521c49244e" path="/var/lib/kubelet/pods/8787e714-bc21-4665-b904-6c521c49244e/volumes" Nov 01 06:10:40.972809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:40.972789 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c8a409-df69-4152-9a60-b57d45130ace" path="/var/lib/kubelet/pods/d6c8a409-df69-4152-9a60-b57d45130ace/volumes" Nov 01 06:10:49.777549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.777516 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr"] Nov 01 06:10:49.777964 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.777952 4425 cpu_manager.go:414] "containerMap: removing container" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" Nov 01 06:10:49.778010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.777968 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" Nov 01 06:10:49.778010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.777994 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ae32a6c5-70c1-421b-90de-b9df11355038" containerName="revoke-keys" Nov 01 06:10:49.778010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.778000 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae32a6c5-70c1-421b-90de-b9df11355038" containerName="revoke-keys" Nov 01 06:10:49.778107 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.778072 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8787e714-bc21-4665-b904-6c521c49244e" containerName="istio-proxy" Nov 01 06:10:49.778107 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.778084 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae32a6c5-70c1-421b-90de-b9df11355038" containerName="revoke-keys" Nov 01 06:10:49.781174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.781157 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.835993 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.835966 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 06:10:49.836187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.836047 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\"" Nov 01 06:10:49.869848 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.869811 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.869884 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.869984 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870023 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a296e9-2e08-439f-96cc-e2c46a96a22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870053 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870153 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870243 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870170 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jgx7\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-kube-api-access-2jgx7\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870243 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870205 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.870359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.870283 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.888965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.888931 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr"] Nov 01 06:10:49.960563 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.960532 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8"] Nov 01 06:10:49.964009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.963993 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:49.970862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.970838 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.970981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.970892 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.970981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.970935 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.970981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.970969 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a296e9-2e08-439f-96cc-e2c46a96a22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.970997 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971038 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971063 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgx7\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-kube-api-access-2jgx7\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971095 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971129 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971372 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971741 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971714 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971765 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a296e9-2e08-439f-96cc-e2c46a96a22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.971904 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.971845 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.972066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.972039 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.973818 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.973788 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:49.975242 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:49.975220 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:50.019080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.018833 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:10:50.019080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.018880 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:10:50.019080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.019014 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:10:50.019080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.019060 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:10:50.021675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.021647 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8"] Nov 01 06:10:50.035474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.035409 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:50.035585 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.035542 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jgx7\" (UniqueName: \"kubernetes.io/projected/61a296e9-2e08-439f-96cc-e2c46a96a22c-kube-api-access-2jgx7\") pod \"maas-default-gateway-openshift-default-58d898bd55-b4bkr\" (UID: \"61a296e9-2e08-439f-96cc-e2c46a96a22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:50.072054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.072020 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz2k\" (UniqueName: \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.072206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.072074 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.072206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.072093 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.094107 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.094078 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:50.172827 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.172788 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz2k\" (UniqueName: \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.173005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.172854 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.173005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.172875 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.173641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.173614 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.175812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.175788 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.235775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.235741 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz2k\" (UniqueName: \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\") pod \"maas-api-revoke-sub-85237e45-9t2g8\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.277216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.277182 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:50.527236 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.527206 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr"] Nov 01 06:10:50.529439 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:10:50.529408 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a296e9_2e08_439f_96cc_e2c46a96a22c.slice/crio-66cf316b8913824aa673c6c9416bf051ea3c7f201254a4c58120794014402208 WatchSource:0}: Error finding container 66cf316b8913824aa673c6c9416bf051ea3c7f201254a4c58120794014402208: Status 404 returned error can't find the container with id 66cf316b8913824aa673c6c9416bf051ea3c7f201254a4c58120794014402208 Nov 01 06:10:50.531889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.531855 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:10:50.531992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.531929 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:10:50.531992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.531967 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:10:50.599127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.599095 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" event={"ID":"61a296e9-2e08-439f-96cc-e2c46a96a22c","Type":"ContainerStarted","Data":"66cf316b8913824aa673c6c9416bf051ea3c7f201254a4c58120794014402208"} Nov 01 06:10:50.656814 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:50.656780 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8"] Nov 01 06:10:50.663105 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:10:50.663072 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e73879_43b0_475b_86f1_8e8badb0d430.slice/crio-f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148 WatchSource:0}: Error finding container f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148: Status 404 returned error can't find the container with id f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148 Nov 01 06:10:51.605857 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:51.605821 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" event={"ID":"55e73879-43b0-475b-86f1-8e8badb0d430","Type":"ContainerStarted","Data":"48dc0a2ce913f161b48ece0f2ee7adbf5aceebdedf9fbb7db8b488b7f4d44bd1"} Nov 01 06:10:51.606356 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:51.605881 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" event={"ID":"55e73879-43b0-475b-86f1-8e8badb0d430","Type":"ContainerStarted","Data":"f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148"} Nov 01 06:10:51.607553 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:51.607524 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" event={"ID":"61a296e9-2e08-439f-96cc-e2c46a96a22c","Type":"ContainerStarted","Data":"7a0ac508e1877ac9074a4593a538e18e9b17bfb542da2da6042e545412475c9a"} Nov 01 06:10:51.731776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:51.731722 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" podStartSLOduration=2.7317055569999997 podStartE2EDuration="2.731705557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:51.729852595 +0000 UTC m=+4066.870441541" watchObservedRunningTime="2026-11-01 06:10:51.731705557 +0000 UTC m=+4066.872294475" Nov 01 06:10:51.863619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:51.863473 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podStartSLOduration=2.863452306 podStartE2EDuration="2.863452306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:51.861579793 +0000 UTC m=+4067.002168731" watchObservedRunningTime="2026-11-01 06:10:51.863452306 +0000 UTC m=+4067.004041227" Nov 01 06:10:52.094461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:52.094417 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:10:52.096169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:52.096143 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:52.096228 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:52.096197 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:52.613765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:52.613727 4425 generic.go:356] "Generic (PLEG): container finished" podID="55e73879-43b0-475b-86f1-8e8badb0d430" containerID="48dc0a2ce913f161b48ece0f2ee7adbf5aceebdedf9fbb7db8b488b7f4d44bd1" exitCode=0 Nov 01 06:10:52.614181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:52.613815 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" event={"ID":"55e73879-43b0-475b-86f1-8e8badb0d430","Type":"ContainerDied","Data":"48dc0a2ce913f161b48ece0f2ee7adbf5aceebdedf9fbb7db8b488b7f4d44bd1"} Nov 01 06:10:53.094707 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.094671 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:53.094879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.094730 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:53.791811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.791785 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:53.909710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.909623 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\") pod \"55e73879-43b0-475b-86f1-8e8badb0d430\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " Nov 01 06:10:53.909873 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.909739 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\") pod \"55e73879-43b0-475b-86f1-8e8badb0d430\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " Nov 01 06:10:53.909873 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.909795 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\" (UniqueName: \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\") pod \"55e73879-43b0-475b-86f1-8e8badb0d430\" (UID: \"55e73879-43b0-475b-86f1-8e8badb0d430\") " Nov 01 06:10:53.910141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.910116 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca" pod "55e73879-43b0-475b-86f1-8e8badb0d430" (UID: "55e73879-43b0-475b-86f1-8e8badb0d430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:53.912043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.912021 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca" pod "55e73879-43b0-475b-86f1-8e8badb0d430" (UID: "55e73879-43b0-475b-86f1-8e8badb0d430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:53.912175 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:53.912154 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k" pod "55e73879-43b0-475b-86f1-8e8badb0d430" (UID: "55e73879-43b0-475b-86f1-8e8badb0d430"). InnerVolumeSpecName "kube-api-access-qfz2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:54.011402 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.011363 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:54.011402 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.011403 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e73879-43b0-475b-86f1-8e8badb0d430-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:54.011623 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.011422 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfz2k\" (UniqueName: \"kubernetes.io/projected/55e73879-43b0-475b-86f1-8e8badb0d430-kube-api-access-qfz2k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:54.095090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.095049 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:54.095322 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.095112 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:54.626190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.626162 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" Nov 01 06:10:54.626190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.626167 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8" event={"ID":"55e73879-43b0-475b-86f1-8e8badb0d430","Type":"ContainerDied","Data":"f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148"} Nov 01 06:10:54.626447 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:54.626221 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7abac21e19d5606e6df33443169131acef5748b2146a9757bd5a8ec64216148" Nov 01 06:10:54.679045 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:10:54.679012 4425 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e73879_43b0_475b_86f1_8e8badb0d430.slice\": RecentStats: unable to find data in memory cache]" Nov 01 06:10:55.095543 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:55.095507 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:55.095986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:55.095566 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:56.095409 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:56.095361 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:56.095621 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:56.095424 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:56.949922 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:10:56.949887 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:10:57.094681 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:57.094649 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:57.094864 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:57.094707 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:10:58.095329 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:58.095292 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 01 06:10:58.095746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:10:58.095350 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 01 06:11:00.136429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:00.136380 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:11:00.137034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:00.136439 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:11:00.878459 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:00.878431 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:11:00.878772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:00.878749 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:11:00.879540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:00.879518 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" Nov 01 06:11:08.413512 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:08.413478 4425 scope.go:122] "RemoveContainer" containerID="3edb8b83b44bc0d86bb87a04b9841157a4b1105e6b856eb6aa89cc76fcbe747e" Nov 01 06:11:08.422654 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:08.422628 4425 scope.go:122] "RemoveContainer" containerID="1b0bf470e6d80def9159527bfe71f3a2a4bb4a981fb12bc5e982d9c1f13c3370" Nov 01 06:11:08.431478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:08.431457 4425 scope.go:122] "RemoveContainer" containerID="c36a60695c49dec585247e2548cc8148956f3c00ccef06f0d423c68dd5ea7379" Nov 01 06:11:13.730880 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:13.730841 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e4d7b1c-a9bf-439c-b355-4149893982d5" containerID="282782d071b6951ff1a4a3c38386fc26e3d58119beaa7f5187d5240237319348" exitCode=1 Nov 01 06:11:13.731433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:13.730927 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" event={"ID":"9e4d7b1c-a9bf-439c-b355-4149893982d5","Type":"ContainerDied","Data":"282782d071b6951ff1a4a3c38386fc26e3d58119beaa7f5187d5240237319348"} Nov 01 06:11:13.731433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:13.731425 4425 scope.go:122] "RemoveContainer" containerID="282782d071b6951ff1a4a3c38386fc26e3d58119beaa7f5187d5240237319348" Nov 01 06:11:13.731862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:13.731842 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:11:14.698471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:14.698437 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 06:11:15.743586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:15.743548 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" event={"ID":"9e4d7b1c-a9bf-439c-b355-4149893982d5","Type":"ContainerStarted","Data":"eeba72cf28aee250c50342e477dd46fd5dcecc3cf0c8b104c28d4deaea82dbff"} Nov 01 06:11:15.744143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:15.743770 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 06:11:26.752120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:26.752031 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-559cb95fbf-czdwv" Nov 01 06:11:26.948172 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:11:26.948123 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:11:44.802357 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.802321 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw"] Nov 01 06:11:44.802963 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.802941 4425 cpu_manager.go:414] "containerMap: removing container" podUID="55e73879-43b0-475b-86f1-8e8badb0d430" containerName="revoke-keys" Nov 01 06:11:44.803062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.802967 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e73879-43b0-475b-86f1-8e8badb0d430" containerName="revoke-keys" Nov 01 06:11:44.803120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.803087 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e73879-43b0-475b-86f1-8e8badb0d430" containerName="revoke-keys" Nov 01 06:11:44.806155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.806134 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:44.861181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.861149 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:11:44.864311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.864279 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw"] Nov 01 06:11:44.900172 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.900141 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:44.900375 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.900212 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnfn\" (UniqueName: \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:44.900375 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.900308 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:44.912026 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.912003 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:11:44.912139 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.912117 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:11:44.912139 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:44.912126 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:11:45.001342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.001302 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.001536 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.001396 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnfn\" (UniqueName: \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.001536 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.001425 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.001999 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.001966 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.003890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.003868 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.064208 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.064169 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnfn\" (UniqueName: \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\") pod \"maas-api-revoke-sub-f904bff9-hskhw\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.119789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.119754 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:45.497952 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.497926 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw"] Nov 01 06:11:45.499799 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:11:45.499770 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2be570_4d25_4115_bd3e_50cb9f40dc1d.slice/crio-15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca WatchSource:0}: Error finding container 15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca: Status 404 returned error can't find the container with id 15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca Nov 01 06:11:45.896778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.896740 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" event={"ID":"0f2be570-4d25-4115-bd3e-50cb9f40dc1d","Type":"ContainerStarted","Data":"65bcd0df089646d8eb1b3160e379d95660c1f11360dc1a5575f5f1c368c7448e"} Nov 01 06:11:45.897205 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:45.896804 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" event={"ID":"0f2be570-4d25-4115-bd3e-50cb9f40dc1d","Type":"ContainerStarted","Data":"15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca"} Nov 01 06:11:46.016459 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:46.016363 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" podStartSLOduration=2.016344215 podStartE2EDuration="2.016344215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:46.014363448 +0000 UTC m=+4121.154952405" watchObservedRunningTime="2026-11-01 06:11:46.016344215 +0000 UTC m=+4121.156933135" Nov 01 06:11:46.903455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:46.903417 4425 generic.go:356] "Generic (PLEG): container finished" podID="0f2be570-4d25-4115-bd3e-50cb9f40dc1d" containerID="65bcd0df089646d8eb1b3160e379d95660c1f11360dc1a5575f5f1c368c7448e" exitCode=0 Nov 01 06:11:46.903951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:46.903505 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" event={"ID":"0f2be570-4d25-4115-bd3e-50cb9f40dc1d","Type":"ContainerDied","Data":"65bcd0df089646d8eb1b3160e379d95660c1f11360dc1a5575f5f1c368c7448e"} Nov 01 06:11:48.042888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.042860 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:48.134051 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.134017 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\" (UniqueName: \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\") pod \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " Nov 01 06:11:48.134234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.134064 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\") pod \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " Nov 01 06:11:48.134234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.134170 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\") pod \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\" (UID: \"0f2be570-4d25-4115-bd3e-50cb9f40dc1d\") " Nov 01 06:11:48.134577 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.134551 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca" pod "0f2be570-4d25-4115-bd3e-50cb9f40dc1d" (UID: "0f2be570-4d25-4115-bd3e-50cb9f40dc1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:11:48.136469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.136448 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn" pod "0f2be570-4d25-4115-bd3e-50cb9f40dc1d" (UID: "0f2be570-4d25-4115-bd3e-50cb9f40dc1d"). InnerVolumeSpecName "kube-api-access-vbnfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:11:48.136539 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.136463 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca" pod "0f2be570-4d25-4115-bd3e-50cb9f40dc1d" (UID: "0f2be570-4d25-4115-bd3e-50cb9f40dc1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:11:48.235636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.235527 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:48.235636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.235568 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:48.235636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.235583 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnfn\" (UniqueName: \"kubernetes.io/projected/0f2be570-4d25-4115-bd3e-50cb9f40dc1d-kube-api-access-vbnfn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:11:48.915393 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.915356 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" event={"ID":"0f2be570-4d25-4115-bd3e-50cb9f40dc1d","Type":"ContainerDied","Data":"15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca"} Nov 01 06:11:48.915586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.915405 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e7025837a7bce301e10d7a955883a5c4355a18158ea48f93e31e5e0c3781ca" Nov 01 06:11:48.915586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:48.915368 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw" Nov 01 06:11:50.046595 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.046559 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r"] Nov 01 06:11:50.097077 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.097038 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5edbba55-d2b4r"] Nov 01 06:11:50.249778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.249721 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q"] Nov 01 06:11:50.250164 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.250151 4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f2be570-4d25-4115-bd3e-50cb9f40dc1d" containerName="revoke-keys" Nov 01 06:11:50.250229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.250165 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2be570-4d25-4115-bd3e-50cb9f40dc1d" containerName="revoke-keys" Nov 01 06:11:50.250299 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.250234 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2be570-4d25-4115-bd3e-50cb9f40dc1d" containerName="revoke-keys" Nov 01 06:11:50.253545 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.253527 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.278284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.278235 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2"] Nov 01 06:11:50.281827 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.281810 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.314472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.314440 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q"] Nov 01 06:11:50.355996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.355828 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.355996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.355892 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.355996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.355923 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.355996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.355946 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356014 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642bdc4b-db0b-4f67-aab4-aa429c34b921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356183 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cb8977-197c-4847-bea0-0d3eea37e625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356339 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356401 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356461 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdn76\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-kube-api-access-fdn76\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356494 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356536 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356671 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356710 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.356982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356744 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.356982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356891 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cb8977-197c-4847-bea0-0d3eea37e625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.356982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.356951 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.357295 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.357017 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.357295 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.357067 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n226\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-kube-api-access-9n226\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.372008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.371969 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2"] Nov 01 06:11:50.458066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458031 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdn76\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-kube-api-access-fdn76\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458073 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458108 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458144 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458179 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458208 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458243 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cb8977-197c-4847-bea0-0d3eea37e625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458319 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458343 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458368 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n226\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-kube-api-access-9n226\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458416 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458446 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458479 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458513 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458569 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458585 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642bdc4b-db0b-4f67-aab4-aa429c34b921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458608 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458642 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cb8977-197c-4847-bea0-0d3eea37e625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458684 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458689 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458744 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.458862 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.458747 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.459562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.459015 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.459562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.459060 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.459562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.459360 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.459562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.459397 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cb8977-197c-4847-bea0-0d3eea37e625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.459562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.459440 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.460276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.460231 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/642bdc4b-db0b-4f67-aab4-aa429c34b921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.460901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.460878 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cb8977-197c-4847-bea0-0d3eea37e625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.461473 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.461441 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.461629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.461609 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.461785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.461768 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cb8977-197c-4847-bea0-0d3eea37e625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.521537 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.521500 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.521700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.521550 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n226\" (UniqueName: \"kubernetes.io/projected/642bdc4b-db0b-4f67-aab4-aa429c34b921-kube-api-access-9n226\") pod \"maas-default-gateway-openshift-default-58d898bd55-8ng9q\" (UID: \"642bdc4b-db0b-4f67-aab4-aa429c34b921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.521700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.521502 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.521700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.521640 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdn76\" (UniqueName: \"kubernetes.io/projected/18cb8977-197c-4847-bea0-0d3eea37e625-kube-api-access-fdn76\") pod \"maas-default-gateway-openshift-default-58d898bd55-kb7j2\" (UID: \"18cb8977-197c-4847-bea0-0d3eea37e625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.567184 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.567109 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:50.595325 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.595300 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:50.971825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:50.971747 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5098b415-11b6-412d-a565-c08bdde6b4ba" path="/var/lib/kubelet/pods/5098b415-11b6-412d-a565-c08bdde6b4ba/volumes" Nov 01 06:11:51.001454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.001422 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q"] Nov 01 06:11:51.004111 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:11:51.004082 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642bdc4b_db0b_4f67_aab4_aa429c34b921.slice/crio-ef61b5da705bf5e61e442361103cc89e1d505ff25e23308acc574d80b6a0d854 WatchSource:0}: Error finding container ef61b5da705bf5e61e442361103cc89e1d505ff25e23308acc574d80b6a0d854: Status 404 returned error can't find the container with id ef61b5da705bf5e61e442361103cc89e1d505ff25e23308acc574d80b6a0d854 Nov 01 06:11:51.005059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.005028 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2"] Nov 01 06:11:51.006410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.006350 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.006497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.006448 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.006497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.006487 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.006611 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:11:51.006588 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cb8977_197c_4847_bea0_0d3eea37e625.slice/crio-1aace7a648745e63c0a966a59c36dc1ffbc9a887745361897defd1b54cf08ed8 WatchSource:0}: Error finding container 1aace7a648745e63c0a966a59c36dc1ffbc9a887745361897defd1b54cf08ed8: Status 404 returned error can't find the container with id 1aace7a648745e63c0a966a59c36dc1ffbc9a887745361897defd1b54cf08ed8 Nov 01 06:11:51.008840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.008816 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.008984 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.008891 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.008984 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.008936 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:11:51.931471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.931432 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" event={"ID":"18cb8977-197c-4847-bea0-0d3eea37e625","Type":"ContainerStarted","Data":"061fd20675c65dcaef2507d51380c6e7a005bb61456c066fabae23589dbba178"} Nov 01 06:11:51.932010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.931494 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" event={"ID":"18cb8977-197c-4847-bea0-0d3eea37e625","Type":"ContainerStarted","Data":"1aace7a648745e63c0a966a59c36dc1ffbc9a887745361897defd1b54cf08ed8"} Nov 01 06:11:51.932885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.932864 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" event={"ID":"642bdc4b-db0b-4f67-aab4-aa429c34b921","Type":"ContainerStarted","Data":"e1c2ca205e45e4dfaaa886cb26d2ce7846ca33e7e0d8ff70b16700e4f27e889b"} Nov 01 06:11:51.933002 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:51.932903 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" event={"ID":"642bdc4b-db0b-4f67-aab4-aa429c34b921","Type":"ContainerStarted","Data":"ef61b5da705bf5e61e442361103cc89e1d505ff25e23308acc574d80b6a0d854"} Nov 01 06:11:52.167277 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.166751 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podStartSLOduration=2.166726105 podStartE2EDuration="2.166726105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:52.104451293 +0000 UTC m=+4127.245040262" watchObservedRunningTime="2026-11-01 06:11:52.166726105 +0000 UTC m=+4127.307315025" Nov 01 06:11:52.267339 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.266835 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podStartSLOduration=2.266813282 podStartE2EDuration="2.266813282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:52.265756359 +0000 UTC m=+4127.406345278" watchObservedRunningTime="2026-11-01 06:11:52.266813282 +0000 UTC m=+4127.407402202" Nov 01 06:11:52.571439 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.571401 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:11:52.571865 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.571840 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:52.572054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.572030 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:52.597591 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.597537 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:11:52.598358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.598332 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:52.598458 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:52.598395 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:53.567806 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:53.567762 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:53.568492 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:53.568455 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:53.596113 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:53.596074 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:53.596311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:53.596142 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:54.586286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:54.584826 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:54.586286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:54.584910 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:54.597034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:54.596995 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:54.597232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:54.597067 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:55.577280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:55.576711 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:55.577280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:55.576777 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:55.595793 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:55.595746 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:55.596226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:55.595812 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:56.568281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:56.568185 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:56.568529 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:56.568313 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:56.596882 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:56.596845 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:56.597429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:56.596893 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:56.998332 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:11:56.998215 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:11:57.567835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:57.567800 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:57.568029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:57.567855 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:57.596206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:57.596167 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:57.596417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:57.596230 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:11:58.568468 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:58.568417 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 01 06:11:58.568897 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:58.568494 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 01 06:11:58.596118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:58.596075 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Nov 01 06:11:58.596324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:11:58.596146 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Nov 01 06:12:00.048469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.048430 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc"] Nov 01 06:12:00.082716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.082677 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d163946-btmxc"] Nov 01 06:12:00.609489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.609446 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:12:00.609691 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.609509 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:12:00.637536 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.637494 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:12:00.637730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.637560 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:12:00.977225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:00.977182 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e82711-b3ce-47c3-bca6-ac7bb754200e" path="/var/lib/kubelet/pods/c9e82711-b3ce-47c3-bca6-ac7bb754200e/volumes" Nov 01 06:12:01.609906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.609869 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:12:01.610477 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.609997 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:12:01.679514 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.679458 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:12:01.679743 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.679548 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:12:01.772524 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.772487 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:12:01.773979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.773953 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:12:01.774679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.774655 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" Nov 01 06:12:01.905073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.904991 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:12:01.905326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.905297 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:12:01.906153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:01.906135 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" Nov 01 06:12:08.470125 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:08.470090 4425 scope.go:122] "RemoveContainer" containerID="c83244fada4333eca9d567ad8e03c4229a5c635d7e31820add4a2e004a5e4e68" Nov 01 06:12:08.478936 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:08.478913 4425 scope.go:122] "RemoveContainer" containerID="c01efd8140006272e87f112d8bfb92d8b16252d4b4862c12dbe0dc8bcb7c39f1" Nov 01 06:12:08.487133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:08.487112 4425 scope.go:122] "RemoveContainer" containerID="074d466772f6dc0be832e07ae4f4ba0cf9096daa9a525ca09253cc1bde1e50d4" Nov 01 06:12:12.049030 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:12.048993 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9"] Nov 01 06:12:12.099705 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:12.099672 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954995f-xrjz9"] Nov 01 06:12:12.973309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:12.973269 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ecf889-64e8-48de-8868-7cb4b8785b3c" path="/var/lib/kubelet/pods/33ecf889-64e8-48de-8868-7cb4b8785b3c/volumes" Nov 01 06:12:18.656003 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.655969 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45"] Nov 01 06:12:18.659767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.659749 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.714729 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.714695 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:12:18.748159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.748113 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.748361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.748185 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.748361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.748308 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llk75\" (UniqueName: \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.765476 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.765448 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:12:18.765476 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.765474 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:12:18.765702 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.765525 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:12:18.767467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.767447 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45"] Nov 01 06:12:18.849272 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.849217 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.849472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.849324 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.849472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.849372 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llk75\" (UniqueName: \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.850040 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.849994 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.851816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.851793 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.911747 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.911669 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llk75\" (UniqueName: \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\") pod \"maas-api-revoke-sub-f03629db-2kr45\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:18.973102 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:18.973072 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:19.344559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:19.344499 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45"] Nov 01 06:12:19.347129 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:12:19.347102 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fde580_8ae9_4942_b76d_ef3a6cfde368.slice/crio-5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee WatchSource:0}: Error finding container 5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee: Status 404 returned error can't find the container with id 5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee Nov 01 06:12:20.202418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:20.202381 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" event={"ID":"53fde580-8ae9-4942-b76d-ef3a6cfde368","Type":"ContainerStarted","Data":"83603f3c17333422bbbf5544cf01553fec3d8e8717e4065f099b424e655d4ad2"} Nov 01 06:12:20.202852 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:20.202438 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" event={"ID":"53fde580-8ae9-4942-b76d-ef3a6cfde368","Type":"ContainerStarted","Data":"5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee"} Nov 01 06:12:20.320597 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:20.320539 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" podStartSLOduration=2.320519912 podStartE2EDuration="2.320519912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:20.319610533 +0000 UTC m=+4155.460199472" watchObservedRunningTime="2026-11-01 06:12:20.320519912 +0000 UTC m=+4155.461108832" Nov 01 06:12:21.209317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:21.209283 4425 generic.go:356] "Generic (PLEG): container finished" podID="53fde580-8ae9-4942-b76d-ef3a6cfde368" containerID="83603f3c17333422bbbf5544cf01553fec3d8e8717e4065f099b424e655d4ad2" exitCode=0 Nov 01 06:12:21.209751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:21.209422 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" event={"ID":"53fde580-8ae9-4942-b76d-ef3a6cfde368","Type":"ContainerDied","Data":"83603f3c17333422bbbf5544cf01553fec3d8e8717e4065f099b424e655d4ad2"} Nov 01 06:12:22.355406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.355381 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:22.384515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.384480 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\") pod \"53fde580-8ae9-4942-b76d-ef3a6cfde368\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " Nov 01 06:12:22.384693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.384547 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\" (UniqueName: \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\") pod \"53fde580-8ae9-4942-b76d-ef3a6cfde368\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " Nov 01 06:12:22.384693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.384650 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\") pod \"53fde580-8ae9-4942-b76d-ef3a6cfde368\" (UID: \"53fde580-8ae9-4942-b76d-ef3a6cfde368\") " Nov 01 06:12:22.385039 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.385014 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca" pod "53fde580-8ae9-4942-b76d-ef3a6cfde368" (UID: "53fde580-8ae9-4942-b76d-ef3a6cfde368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:12:22.386786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.386765 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca" pod "53fde580-8ae9-4942-b76d-ef3a6cfde368" (UID: "53fde580-8ae9-4942-b76d-ef3a6cfde368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:12:22.386871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.386858 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75" pod "53fde580-8ae9-4942-b76d-ef3a6cfde368" (UID: "53fde580-8ae9-4942-b76d-ef3a6cfde368"). InnerVolumeSpecName "kube-api-access-llk75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:22.486305 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.486200 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:22.486305 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.486228 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fde580-8ae9-4942-b76d-ef3a6cfde368-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:22.486305 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:22.486238 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llk75\" (UniqueName: \"kubernetes.io/projected/53fde580-8ae9-4942-b76d-ef3a6cfde368-kube-api-access-llk75\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:23.045662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:23.045620 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87"] Nov 01 06:12:23.097798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:23.097755 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0ce180-88n87"] Nov 01 06:12:23.220450 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:23.220417 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" Nov 01 06:12:23.220631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:23.220454 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45" event={"ID":"53fde580-8ae9-4942-b76d-ef3a6cfde368","Type":"ContainerDied","Data":"5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee"} Nov 01 06:12:23.220631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:23.220534 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e5581366886a00706490899d873873b5bbac401791ce185f44755b8c76e97ee" Nov 01 06:12:24.972070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:24.972034 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b10d240-ed5e-450c-b8cc-f59ac95db8e4" path="/var/lib/kubelet/pods/2b10d240-ed5e-450c-b8cc-f59ac95db8e4/volumes" Nov 01 06:12:26.947556 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:12:26.947459 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:12:38.519978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.519945 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77"] Nov 01 06:12:38.520448 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.520432 4425 cpu_manager.go:414] "containerMap: removing container" podUID="53fde580-8ae9-4942-b76d-ef3a6cfde368" containerName="revoke-keys" Nov 01 06:12:38.520499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.520450 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fde580-8ae9-4942-b76d-ef3a6cfde368" containerName="revoke-keys" Nov 01 06:12:38.520536 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.520526 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fde580-8ae9-4942-b76d-ef3a6cfde368" containerName="revoke-keys" Nov 01 06:12:38.523709 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.523690 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.581569 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.581529 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77"] Nov 01 06:12:38.629632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.629597 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:12:38.629632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.629610 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:12:38.629632 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.629626 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:12:38.629884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.629641 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:12:38.644982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.644953 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.645174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.645121 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4xg\" (UniqueName: \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.645226 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.645174 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.746499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.746463 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.746708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.746513 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.746708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.746595 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4xg\" (UniqueName: \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.747212 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.747175 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.749057 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.749036 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.808969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.808890 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4xg\" (UniqueName: \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\") pod \"maas-api-revoke-sub-0165d7d2-h6v77\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:38.837403 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:38.837368 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:39.212206 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:39.212179 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77"] Nov 01 06:12:39.214090 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:12:39.214066 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6298c8_379c_40c0_8873_9f34c713dc3f.slice/crio-7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9 WatchSource:0}: Error finding container 7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9: Status 404 returned error can't find the container with id 7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9 Nov 01 06:12:39.323210 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:39.323175 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" event={"ID":"2c6298c8-379c-40c0-8873-9f34c713dc3f","Type":"ContainerStarted","Data":"7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9"} Nov 01 06:12:40.330116 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:40.330077 4425 generic.go:356] "Generic (PLEG): container finished" podID="2c6298c8-379c-40c0-8873-9f34c713dc3f" containerID="becc46bf6ec77e985c285a73db1185b2d60f45e4f346bb7294db71fd2aa51089" exitCode=0 Nov 01 06:12:40.330116 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:40.330112 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" event={"ID":"2c6298c8-379c-40c0-8873-9f34c713dc3f","Type":"ContainerDied","Data":"becc46bf6ec77e985c285a73db1185b2d60f45e4f346bb7294db71fd2aa51089"} Nov 01 06:12:41.469451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.469425 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:41.574599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.574561 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\") pod \"2c6298c8-379c-40c0-8873-9f34c713dc3f\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " Nov 01 06:12:41.574599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.574602 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\") pod \"2c6298c8-379c-40c0-8873-9f34c713dc3f\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " Nov 01 06:12:41.574889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.574656 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\" (UniqueName: \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\") pod \"2c6298c8-379c-40c0-8873-9f34c713dc3f\" (UID: \"2c6298c8-379c-40c0-8873-9f34c713dc3f\") " Nov 01 06:12:41.575040 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.575018 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca" pod "2c6298c8-379c-40c0-8873-9f34c713dc3f" (UID: "2c6298c8-379c-40c0-8873-9f34c713dc3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:12:41.576964 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.576941 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca" pod "2c6298c8-379c-40c0-8873-9f34c713dc3f" (UID: "2c6298c8-379c-40c0-8873-9f34c713dc3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:12:41.577049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.576982 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg" pod "2c6298c8-379c-40c0-8873-9f34c713dc3f" (UID: "2c6298c8-379c-40c0-8873-9f34c713dc3f"). InnerVolumeSpecName "kube-api-access-gp4xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:41.676195 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.676111 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:41.676195 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.676144 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6298c8-379c-40c0-8873-9f34c713dc3f-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:41.676195 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:41.676155 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4xg\" (UniqueName: \"kubernetes.io/projected/2c6298c8-379c-40c0-8873-9f34c713dc3f-kube-api-access-gp4xg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:42.341598 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:42.341571 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" Nov 01 06:12:42.341785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:42.341566 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77" event={"ID":"2c6298c8-379c-40c0-8873-9f34c713dc3f","Type":"ContainerDied","Data":"7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9"} Nov 01 06:12:42.341785 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:42.341702 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6403e2a5728b2cc722e429a217be81b3b54f55893afc74cff2067869074ec9" Nov 01 06:12:50.253567 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.253479 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm"] Nov 01 06:12:50.254196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.253898 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2c6298c8-379c-40c0-8873-9f34c713dc3f" containerName="revoke-keys" Nov 01 06:12:50.254196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.253909 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6298c8-379c-40c0-8873-9f34c713dc3f" containerName="revoke-keys" Nov 01 06:12:50.254196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.253970 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6298c8-379c-40c0-8873-9f34c713dc3f" containerName="revoke-keys" Nov 01 06:12:50.257162 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.257142 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.312644 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.312612 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:12:50.315666 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.315630 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm"] Nov 01 06:12:50.363461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.363421 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:12:50.363461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.363441 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:12:50.363697 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.363487 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:12:50.373981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.373953 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.374072 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.373997 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.374144 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.374120 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx6g\" (UniqueName: \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.475707 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.475666 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.475908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.475717 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.475908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.475758 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx6g\" (UniqueName: \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.476568 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.476543 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.478248 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.478225 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.538548 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.538471 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx6g\" (UniqueName: \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\") pod \"maas-api-revoke-sub-6b19338f-jpllm\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.570410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.570381 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:50.950972 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:50.950945 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm"] Nov 01 06:12:50.952867 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:12:50.952836 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeda806_16da_44c7_868c_122b5e5af7e5.slice/crio-be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877 WatchSource:0}: Error finding container be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877: Status 404 returned error can't find the container with id be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877 Nov 01 06:12:51.393872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:51.393836 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" event={"ID":"5eeda806-16da-44c7-868c-122b5e5af7e5","Type":"ContainerStarted","Data":"9e7061421bf2db671536c54f97153d00a8e3f5a0601803dd9be5212eed3e6b06"} Nov 01 06:12:51.393872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:51.393892 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" event={"ID":"5eeda806-16da-44c7-868c-122b5e5af7e5","Type":"ContainerStarted","Data":"be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877"} Nov 01 06:12:51.514629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:51.514531 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" podStartSLOduration=1.514515105 podStartE2EDuration="1.514515105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:51.513448846 +0000 UTC m=+4186.654037778" watchObservedRunningTime="2026-11-01 06:12:51.514515105 +0000 UTC m=+4186.655104023" Nov 01 06:12:52.400629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:52.400592 4425 generic.go:356] "Generic (PLEG): container finished" podID="5eeda806-16da-44c7-868c-122b5e5af7e5" containerID="9e7061421bf2db671536c54f97153d00a8e3f5a0601803dd9be5212eed3e6b06" exitCode=0 Nov 01 06:12:52.401112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:52.400646 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" event={"ID":"5eeda806-16da-44c7-868c-122b5e5af7e5","Type":"ContainerDied","Data":"9e7061421bf2db671536c54f97153d00a8e3f5a0601803dd9be5212eed3e6b06"} Nov 01 06:12:53.539140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.539114 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:53.607556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.607526 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\") pod \"5eeda806-16da-44c7-868c-122b5e5af7e5\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " Nov 01 06:12:53.607728 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.607627 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\") pod \"5eeda806-16da-44c7-868c-122b5e5af7e5\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " Nov 01 06:12:53.607728 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.607690 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\" (UniqueName: \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\") pod \"5eeda806-16da-44c7-868c-122b5e5af7e5\" (UID: \"5eeda806-16da-44c7-868c-122b5e5af7e5\") " Nov 01 06:12:53.607914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.607886 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca" pod "5eeda806-16da-44c7-868c-122b5e5af7e5" (UID: "5eeda806-16da-44c7-868c-122b5e5af7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:12:53.609873 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.609846 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca" pod "5eeda806-16da-44c7-868c-122b5e5af7e5" (UID: "5eeda806-16da-44c7-868c-122b5e5af7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:12:53.610024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.610006 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g" pod "5eeda806-16da-44c7-868c-122b5e5af7e5" (UID: "5eeda806-16da-44c7-868c-122b5e5af7e5"). InnerVolumeSpecName "kube-api-access-bfx6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:12:53.708748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.708665 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfx6g\" (UniqueName: \"kubernetes.io/projected/5eeda806-16da-44c7-868c-122b5e5af7e5-kube-api-access-bfx6g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:53.708748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.708696 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:53.708748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:53.708706 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeda806-16da-44c7-868c-122b5e5af7e5-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:12:54.412639 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:54.412604 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" event={"ID":"5eeda806-16da-44c7-868c-122b5e5af7e5","Type":"ContainerDied","Data":"be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877"} Nov 01 06:12:54.412830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:54.412658 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be22d13f0eb33b52eb43e486a15b525bfa8fd97903c3ede2bd53ee24d1d74877" Nov 01 06:12:54.412830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:12:54.412614 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm" Nov 01 06:12:56.947537 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:12:56.947499 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:13:05.417865 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.417830 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:13:05.418369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.418247 4425 cpu_manager.go:414] "containerMap: removing container" podUID="5eeda806-16da-44c7-868c-122b5e5af7e5" containerName="revoke-keys" Nov 01 06:13:05.418369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.418276 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeda806-16da-44c7-868c-122b5e5af7e5" containerName="revoke-keys" Nov 01 06:13:05.418369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.418336 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeda806-16da-44c7-868c-122b5e5af7e5" containerName="revoke-keys" Nov 01 06:13:05.421647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.421629 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.525906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.525874 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-25304531-kserve-self-signed-certs\"" Nov 01 06:13:05.528098 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.528076 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:13:05.531666 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531637 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.531783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531691 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jnj\" (UniqueName: \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.531835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531792 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.531893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531845 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.531950 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531910 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.531995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.531969 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.632857 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.632825 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jnj\" (UniqueName: \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.632876 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.632917 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.632937 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.632956 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.633020 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633449 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.633427 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.633504 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.633590 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.633502 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.635674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.635643 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.635929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.635913 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.695242 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.695166 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jnj\" (UniqueName: \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\") pod \"e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:05.734874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:05.734839 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:06.110975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:06.110949 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:13:06.113086 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:13:06.113049 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6b26a0_1f6a_4197_8d01_558f20e99e98.slice/crio-308522a61d4a491297de299871ca4194e54269426c54fe9781e228027d53d78f WatchSource:0}: Error finding container 308522a61d4a491297de299871ca4194e54269426c54fe9781e228027d53d78f: Status 404 returned error can't find the container with id 308522a61d4a491297de299871ca4194e54269426c54fe9781e228027d53d78f Nov 01 06:13:06.506241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:06.506149 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" event={"ID":"ce6b26a0-1f6a-4197-8d01-558f20e99e98","Type":"ContainerStarted","Data":"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d"} Nov 01 06:13:06.506241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:06.506190 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" event={"ID":"ce6b26a0-1f6a-4197-8d01-558f20e99e98","Type":"ContainerStarted","Data":"308522a61d4a491297de299871ca4194e54269426c54fe9781e228027d53d78f"} Nov 01 06:13:06.625223 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:06.625167 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" podStartSLOduration=1.625152417 podStartE2EDuration="1.625152417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:06.624093434 +0000 UTC m=+4201.764682373" watchObservedRunningTime="2026-11-01 06:13:06.625152417 +0000 UTC m=+4201.765741335" Nov 01 06:13:06.735562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:06.735522 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:08.515288 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:08.515238 4425 scope.go:122] "RemoveContainer" containerID="35faa3492bba683d9e3f11c290eda723060148901bc533a5ea15d68f6d4ab375" Nov 01 06:13:08.524305 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:08.524281 4425 scope.go:122] "RemoveContainer" containerID="cbbe4be6e9b4aba7fb2f11c98ef3a682f3b3fe620b18e7d7074ad6fd61788860" Nov 01 06:13:15.735294 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:15.735230 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:15.748176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:15.748145 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:16.570441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:16.570412 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:13:26.946389 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:13:26.946350 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:13:56.947596 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:13:56.947557 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:13:59.964063 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:59.964030 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm"] Nov 01 06:13:59.967996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:13:59.967978 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.019181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.019143 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.019394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.019321 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.019440 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.019419 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbjc\" (UniqueName: \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.024544 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.024512 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:14:00.075392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.075356 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:14:00.075392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.075385 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:14:00.075681 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.075459 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:14:00.077428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.077408 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm"] Nov 01 06:14:00.120193 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.120161 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.120429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.120295 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbjc\" (UniqueName: \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.120429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.120337 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.120949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.120927 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.122895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.122873 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.182844 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.182814 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbjc\" (UniqueName: \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\") pod \"maas-api-revoke-sub-06434dd9-bnxnm\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.281838 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.281743 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:00.657657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.657629 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm"] Nov 01 06:14:00.659546 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:14:00.659519 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337e172d_73b4_44ef_a890_9a1b88dd5ba6.slice/crio-ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f WatchSource:0}: Error finding container ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f: Status 404 returned error can't find the container with id ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f Nov 01 06:14:00.862246 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:00.862212 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" event={"ID":"337e172d-73b4-44ef-a890-9a1b88dd5ba6","Type":"ContainerStarted","Data":"ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f"} Nov 01 06:14:01.868833 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:01.868797 4425 generic.go:356] "Generic (PLEG): container finished" podID="337e172d-73b4-44ef-a890-9a1b88dd5ba6" containerID="e754184f42de0136bc93798f8b0492a08a574a87b4709ea0cd5271964f58b88e" exitCode=0 Nov 01 06:14:01.869317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:01.868888 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" event={"ID":"337e172d-73b4-44ef-a890-9a1b88dd5ba6","Type":"ContainerDied","Data":"e754184f42de0136bc93798f8b0492a08a574a87b4709ea0cd5271964f58b88e"} Nov 01 06:14:03.006700 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.006672 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:03.152654 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.152522 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\" (UniqueName: \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\") pod \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " Nov 01 06:14:03.152854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.152667 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\") pod \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " Nov 01 06:14:03.152854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.152728 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\") pod \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\" (UID: \"337e172d-73b4-44ef-a890-9a1b88dd5ba6\") " Nov 01 06:14:03.153170 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.153137 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca" pod "337e172d-73b4-44ef-a890-9a1b88dd5ba6" (UID: "337e172d-73b4-44ef-a890-9a1b88dd5ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:03.154915 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.154887 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc" pod "337e172d-73b4-44ef-a890-9a1b88dd5ba6" (UID: "337e172d-73b4-44ef-a890-9a1b88dd5ba6"). InnerVolumeSpecName "kube-api-access-4wbjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:03.155015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.154915 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca" pod "337e172d-73b4-44ef-a890-9a1b88dd5ba6" (UID: "337e172d-73b4-44ef-a890-9a1b88dd5ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:03.253919 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.253886 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbjc\" (UniqueName: \"kubernetes.io/projected/337e172d-73b4-44ef-a890-9a1b88dd5ba6-kube-api-access-4wbjc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:03.253919 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.253917 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:03.254135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.253930 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337e172d-73b4-44ef-a890-9a1b88dd5ba6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:03.879806 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.879776 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" Nov 01 06:14:03.880021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.879775 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm" event={"ID":"337e172d-73b4-44ef-a890-9a1b88dd5ba6","Type":"ContainerDied","Data":"ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f"} Nov 01 06:14:03.880021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:03.879905 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2cc832fcd43462467fbd8c750b1b5f5ef964b2c4efa29a1d34665787589d9f" Nov 01 06:14:08.548332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:08.548296 4425 scope.go:122] "RemoveContainer" containerID="52e619dc9e9987b6fbff01d9f379c539f5843be6beae8f86de1393a76cb60937" Nov 01 06:14:08.557346 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:08.557326 4425 scope.go:122] "RemoveContainer" containerID="6f4077c6d2f022b25a4e93f2a077b9b8ed383437804c9bc42bcfd5d309b18c30" Nov 01 06:14:12.960129 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:12.960093 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:14:18.461222 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.461121 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x"] Nov 01 06:14:18.461846 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.461821 4425 cpu_manager.go:414] "containerMap: removing container" podUID="337e172d-73b4-44ef-a890-9a1b88dd5ba6" containerName="revoke-keys" Nov 01 06:14:18.461930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.461849 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337e172d-73b4-44ef-a890-9a1b88dd5ba6" containerName="revoke-keys" Nov 01 06:14:18.462003 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.461959 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="337e172d-73b4-44ef-a890-9a1b88dd5ba6" containerName="revoke-keys" Nov 01 06:14:18.465578 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.465536 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.521510 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.521478 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x"] Nov 01 06:14:18.568713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.568681 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:14:18.568930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.568758 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:14:18.568930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.568823 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:14:18.570892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.570858 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:14:18.605885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.605845 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6xf4\" (UniqueName: \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.606106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.605903 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.606106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.605987 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.706917 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.706875 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6xf4\" (UniqueName: \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.707134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.706927 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.707134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.706970 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.707943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.707852 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.709826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.709804 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.770307 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.770187 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6xf4\" (UniqueName: \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\") pod \"maas-api-revoke-sub-e651d27e-99m6x\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:18.779892 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:18.779851 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:19.185235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:19.185176 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x"] Nov 01 06:14:20.011767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:20.011659 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" event={"ID":"d7f3a6ca-f5cd-44af-9d98-6f7394031e10","Type":"ContainerStarted","Data":"d6dc9c177b91187f8727943cc14dc3564d260bc5812b8af3796c1e65113dddee"} Nov 01 06:14:20.012408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:20.012361 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" event={"ID":"d7f3a6ca-f5cd-44af-9d98-6f7394031e10","Type":"ContainerStarted","Data":"2346a76a792e03edc915e4c66e4cafb78d2cb7805aecafa0159cc1d2f38d51a2"} Nov 01 06:14:20.133726 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:20.133670 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" podStartSLOduration=2.133654459 podStartE2EDuration="2.133654459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:20.131552927 +0000 UTC m=+4275.272141865" watchObservedRunningTime="2026-11-01 06:14:20.133654459 +0000 UTC m=+4275.274243373" Nov 01 06:14:21.025552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:21.025513 4425 generic.go:356] "Generic (PLEG): container finished" podID="d7f3a6ca-f5cd-44af-9d98-6f7394031e10" containerID="d6dc9c177b91187f8727943cc14dc3564d260bc5812b8af3796c1e65113dddee" exitCode=0 Nov 01 06:14:21.026005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:21.025598 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" event={"ID":"d7f3a6ca-f5cd-44af-9d98-6f7394031e10","Type":"ContainerDied","Data":"d6dc9c177b91187f8727943cc14dc3564d260bc5812b8af3796c1e65113dddee"} Nov 01 06:14:22.163596 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.163570 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:22.250567 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.250529 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\") pod \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " Nov 01 06:14:22.250786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.250660 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\" (UniqueName: \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\") pod \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " Nov 01 06:14:22.250786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.250688 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\") pod \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\" (UID: \"d7f3a6ca-f5cd-44af-9d98-6f7394031e10\") " Nov 01 06:14:22.250944 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.250916 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca" pod "d7f3a6ca-f5cd-44af-9d98-6f7394031e10" (UID: "d7f3a6ca-f5cd-44af-9d98-6f7394031e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:14:22.251103 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.251083 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:22.252992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.252965 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca" pod "d7f3a6ca-f5cd-44af-9d98-6f7394031e10" (UID: "d7f3a6ca-f5cd-44af-9d98-6f7394031e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:22.253127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.253082 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4" pod "d7f3a6ca-f5cd-44af-9d98-6f7394031e10" (UID: "d7f3a6ca-f5cd-44af-9d98-6f7394031e10"). InnerVolumeSpecName "kube-api-access-c6xf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:22.351706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.351672 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6xf4\" (UniqueName: \"kubernetes.io/projected/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-kube-api-access-c6xf4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:22.351706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:22.351703 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f3a6ca-f5cd-44af-9d98-6f7394031e10-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:23.037432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:23.037399 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" event={"ID":"d7f3a6ca-f5cd-44af-9d98-6f7394031e10","Type":"ContainerDied","Data":"2346a76a792e03edc915e4c66e4cafb78d2cb7805aecafa0159cc1d2f38d51a2"} Nov 01 06:14:23.037617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:23.037442 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x" Nov 01 06:14:23.037617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:23.037450 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2346a76a792e03edc915e4c66e4cafb78d2cb7805aecafa0159cc1d2f38d51a2" Nov 01 06:14:26.947134 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:14:26.947099 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:14:27.982958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:27.982893 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" podUID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" containerName="main" containerID="cri-o://3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d" gracePeriod=45 Nov 01 06:14:28.234478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.234392 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:14:28.308611 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308582 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.308804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308680 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.308804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308714 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.308804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308751 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\" (UniqueName: \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.308804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308781 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.309030 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.308810 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\") pod \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\" (UID: \"ce6b26a0-1f6a-4197-8d01-558f20e99e98\") " Nov 01 06:14:28.309112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309005 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:28.309112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309042 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:28.309112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309069 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:28.309235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309201 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:28.309235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309222 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:28.309354 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.309237 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:28.311127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.311106 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:14:28.311210 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.311138 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:28.311282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.311220 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj" pod "ce6b26a0-1f6a-4197-8d01-558f20e99e98" (UID: "ce6b26a0-1f6a-4197-8d01-558f20e99e98"). InnerVolumeSpecName "kube-api-access-h8jnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:28.410535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.410500 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8jnj\" (UniqueName: \"kubernetes.io/projected/ce6b26a0-1f6a-4197-8d01-558f20e99e98-kube-api-access-h8jnj\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:28.410535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.410530 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6b26a0-1f6a-4197-8d01-558f20e99e98-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:28.410535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:28.410540 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6b26a0-1f6a-4197-8d01-558f20e99e98-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:29.069905 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.069871 4425 generic.go:356] "Generic (PLEG): container finished" podID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" containerID="3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d" exitCode=0 Nov 01 06:14:29.069905 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.069910 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" Nov 01 06:14:29.070499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.069955 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" event={"ID":"ce6b26a0-1f6a-4197-8d01-558f20e99e98","Type":"ContainerDied","Data":"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d"} Nov 01 06:14:29.070499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.070005 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8" event={"ID":"ce6b26a0-1f6a-4197-8d01-558f20e99e98","Type":"ContainerDied","Data":"308522a61d4a491297de299871ca4194e54269426c54fe9781e228027d53d78f"} Nov 01 06:14:29.070499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.070031 4425 scope.go:122] "RemoveContainer" containerID="3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d" Nov 01 06:14:29.078951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.078930 4425 scope.go:122] "RemoveContainer" containerID="3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d" Nov 01 06:14:29.079245 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:14:29.079225 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d\": container with ID starting with 3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d not found: ID does not exist" containerID="3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d" Nov 01 06:14:29.079332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.079272 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d"} err="failed to get container status \"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d\": rpc error: code = NotFound desc = could not find container \"3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d\": container with ID starting with 3f157bcf7b868dcc73ea739ff1d647665477991087b9d29d9e812dd7c261af8d not found: ID does not exist" Nov 01 06:14:29.236444 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.236361 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:14:29.287285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:29.287242 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-5548ccf5c4-zzxc8"] Nov 01 06:14:30.971978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:30.971942 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" path="/var/lib/kubelet/pods/ce6b26a0-1f6a-4197-8d01-558f20e99e98/volumes" Nov 01 06:14:39.706807 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.706770 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707182 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d7f3a6ca-f5cd-44af-9d98-6f7394031e10" containerName="revoke-keys" Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707192 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f3a6ca-f5cd-44af-9d98-6f7394031e10" containerName="revoke-keys" Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707217 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" containerName="main" Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707223 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" containerName="main" Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707300 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f3a6ca-f5cd-44af-9d98-6f7394031e10" containerName="revoke-keys" Nov 01 06:14:39.707332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.707312 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6b26a0-1f6a-4197-8d01-558f20e99e98" containerName="main" Nov 01 06:14:39.711757 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.711740 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.768592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.768561 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:14:39.771942 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.771907 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.772116 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.771982 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.772116 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.772056 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.772290 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.772160 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdmz\" (UniqueName: \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.772290 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.772210 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.772423 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.772370 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.792422 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.792388 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f6cbc132-kserve-self-signed-certs\"" Nov 01 06:14:39.873121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873059 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873165 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873217 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873277 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873318 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdmz\" (UniqueName: \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873363 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873351 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873667 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873557 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873667 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873572 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.873801 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.873781 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.875936 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.875910 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.876149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.876132 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:39.936169 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:39.936141 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdmz\" (UniqueName: \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\") pod \"e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:40.024150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:40.024048 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:40.406211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:40.406180 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:14:40.407842 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:14:40.407814 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod193d32db_a35f_41fe_a727_b5925d24343f.slice/crio-6b9cbd9b5e6ebb5121bf71b92eef5b04b01abd7325b8219ce489cd32630b711a WatchSource:0}: Error finding container 6b9cbd9b5e6ebb5121bf71b92eef5b04b01abd7325b8219ce489cd32630b711a: Status 404 returned error can't find the container with id 6b9cbd9b5e6ebb5121bf71b92eef5b04b01abd7325b8219ce489cd32630b711a Nov 01 06:14:41.152816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:41.152769 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" event={"ID":"193d32db-a35f-41fe-a727-b5925d24343f","Type":"ContainerStarted","Data":"fb4351aa702a2a75beebb3345490d9733ac8fdc048d5035119acaf076be1b20e"} Nov 01 06:14:41.152816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:41.152811 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" event={"ID":"193d32db-a35f-41fe-a727-b5925d24343f","Type":"ContainerStarted","Data":"6b9cbd9b5e6ebb5121bf71b92eef5b04b01abd7325b8219ce489cd32630b711a"} Nov 01 06:14:41.272146 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:41.272094 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" podStartSLOduration=2.272079503 podStartE2EDuration="2.272079503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:41.270679169 +0000 UTC m=+4296.411268105" watchObservedRunningTime="2026-11-01 06:14:41.272079503 +0000 UTC m=+4296.412668472" Nov 01 06:14:42.024988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:42.024950 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:46.921467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:46.921419 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:14:46.922011 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:46.921498 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:14:50.024236 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:50.024195 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:50.036946 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:50.036919 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:50.248579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:14:50.248553 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:14:56.946397 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:14:56.946352 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:15:00.178539 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.178503 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:00.182457 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.182431 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.236289 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.236247 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:00.236536 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.236425 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:15:00.236668 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.236585 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:00.241477 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.241446 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.241607 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.241498 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfsn\" (UniqueName: \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.285469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.285435 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:00.342690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.342651 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.342690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.342691 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfsn\" (UniqueName: \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.343315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.343294 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.405032 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.404999 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfsn\" (UniqueName: \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\") pod \"maas-api-key-cleanup-29891895-fcjcf\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.495849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.495765 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:00.870353 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:00.870330 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:00.872299 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:15:00.872244 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2970c75_769e_4f3d_b61c_06dd0dfaed7c.slice/crio-dc29b308febabba93126dfcf19b80a296cb8a4e6d1226b27887833ec99ef0f3b WatchSource:0}: Error finding container dc29b308febabba93126dfcf19b80a296cb8a4e6d1226b27887833ec99ef0f3b: Status 404 returned error can't find the container with id dc29b308febabba93126dfcf19b80a296cb8a4e6d1226b27887833ec99ef0f3b Nov 01 06:15:01.304032 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:01.303982 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerStarted","Data":"86a8a8e60066cf56a0c0db0269134e3ba0355d46c850bf1abca91b6b6ca497e1"} Nov 01 06:15:01.304464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:01.304050 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerStarted","Data":"dc29b308febabba93126dfcf19b80a296cb8a4e6d1226b27887833ec99ef0f3b"} Nov 01 06:15:01.420908 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:01.420804 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" podStartSLOduration=1.420786029 podStartE2EDuration="1.420786029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.420402684 +0000 UTC m=+4316.560991615" watchObservedRunningTime="2026-11-01 06:15:01.420786029 +0000 UTC m=+4316.561374949" Nov 01 06:15:02.310616 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:02.310583 4425 generic.go:356] "Generic (PLEG): container finished" podID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerID="86a8a8e60066cf56a0c0db0269134e3ba0355d46c850bf1abca91b6b6ca497e1" exitCode=60 Nov 01 06:15:02.311025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:02.310669 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerDied","Data":"86a8a8e60066cf56a0c0db0269134e3ba0355d46c850bf1abca91b6b6ca497e1"} Nov 01 06:15:02.311025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:02.310873 4425 scope.go:122] "RemoveContainer" containerID="86a8a8e60066cf56a0c0db0269134e3ba0355d46c850bf1abca91b6b6ca497e1" Nov 01 06:15:03.317675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:03.317634 4425 generic.go:356] "Generic (PLEG): container finished" podID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerID="657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda" exitCode=60 Nov 01 06:15:03.318124 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:03.317719 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerDied","Data":"657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda"} Nov 01 06:15:03.318124 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:03.317776 4425 scope.go:122] "RemoveContainer" containerID="86a8a8e60066cf56a0c0db0269134e3ba0355d46c850bf1abca91b6b6ca497e1" Nov 01 06:15:03.318124 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:03.317999 4425 scope.go:122] "RemoveContainer" containerID="657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda" Nov 01 06:15:03.318341 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:15:03.318321 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-fcjcf_odh-ai-gateway-infra(f2970c75-769e-4f3d-b61c-06dd0dfaed7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" Nov 01 06:15:04.325234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:04.325206 4425 scope.go:122] "RemoveContainer" containerID="657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda" Nov 01 06:15:04.325662 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:15:04.325461 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-fcjcf_odh-ai-gateway-infra(f2970c75-769e-4f3d-b61c-06dd0dfaed7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" Nov 01 06:15:14.975833 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:14.975801 4425 scope.go:122] "RemoveContainer" containerID="657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda" Nov 01 06:15:15.414134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:15.414092 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerStarted","Data":"0ac04e6f13592b503a92fa91b172df586d8925538b3c24c3bb993798a88d7321"} Nov 01 06:15:16.421110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:16.421068 4425 generic.go:356] "Generic (PLEG): container finished" podID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerID="0ac04e6f13592b503a92fa91b172df586d8925538b3c24c3bb993798a88d7321" exitCode=60 Nov 01 06:15:16.421110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:16.421114 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerDied","Data":"0ac04e6f13592b503a92fa91b172df586d8925538b3c24c3bb993798a88d7321"} Nov 01 06:15:16.421628 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:16.421161 4425 scope.go:122] "RemoveContainer" containerID="657c94060266b1dc23fe64a53ba68633bc7c435075951de98dcc09a619882eda" Nov 01 06:15:16.421628 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:16.421519 4425 scope.go:122] "RemoveContainer" containerID="0ac04e6f13592b503a92fa91b172df586d8925538b3c24c3bb993798a88d7321" Nov 01 06:15:16.421785 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:15:16.421763 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891895-fcjcf_odh-ai-gateway-infra(f2970c75-769e-4f3d-b61c-06dd0dfaed7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" Nov 01 06:15:16.547033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:16.546997 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:17.582688 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.582655 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:17.660659 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.660617 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\" (UniqueName: \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\") pod \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " Nov 01 06:15:17.660872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.660764 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\") pod \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\" (UID: \"f2970c75-769e-4f3d-b61c-06dd0dfaed7c\") " Nov 01 06:15:17.661161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.661139 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca" pod "f2970c75-769e-4f3d-b61c-06dd0dfaed7c" (UID: "f2970c75-769e-4f3d-b61c-06dd0dfaed7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:17.663080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.663060 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn" pod "f2970c75-769e-4f3d-b61c-06dd0dfaed7c" (UID: "f2970c75-769e-4f3d-b61c-06dd0dfaed7c"). InnerVolumeSpecName "kube-api-access-ktfsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:17.761924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.761829 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:17.761924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:17.761859 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfsn\" (UniqueName: \"kubernetes.io/projected/f2970c75-769e-4f3d-b61c-06dd0dfaed7c-kube-api-access-ktfsn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:18.434162 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.434129 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" Nov 01 06:15:18.434381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.434132 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf" event={"ID":"f2970c75-769e-4f3d-b61c-06dd0dfaed7c","Type":"ContainerDied","Data":"dc29b308febabba93126dfcf19b80a296cb8a4e6d1226b27887833ec99ef0f3b"} Nov 01 06:15:18.434381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.434298 4425 scope.go:122] "RemoveContainer" containerID="0ac04e6f13592b503a92fa91b172df586d8925538b3c24c3bb993798a88d7321" Nov 01 06:15:18.601111 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.601061 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:18.604931 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.604899 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891895-fcjcf"] Nov 01 06:15:18.972354 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:18.972321 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" path="/var/lib/kubelet/pods/f2970c75-769e-4f3d-b61c-06dd0dfaed7c/volumes" Nov 01 06:15:26.946496 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:15:26.946451 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:15:34.675827 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.675793 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml"] Nov 01 06:15:34.676265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676181 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676192 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676206 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676211 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676310 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676323 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.676413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.676329 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:15:34.681036 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.681018 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.734559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.734535 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:34.785174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.785132 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:15:34.785405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.785207 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:34.785405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.785248 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:15:34.787431 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.787403 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml"] Nov 01 06:15:34.847598 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.847565 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.847808 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.847615 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg25t\" (UniqueName: \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.847808 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.847740 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.949091 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.949002 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.949091 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.949054 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg25t\" (UniqueName: \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.949318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.949111 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.949730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.949707 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:34.951673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:34.951650 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:35.013999 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:35.013968 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg25t\" (UniqueName: \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\") pod \"maas-api-revoke-sub-452b6d7b-kzhml\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:35.293805 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:35.293717 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:35.668934 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:35.668906 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml"] Nov 01 06:15:35.670003 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:15:35.669979 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e0b7c0_d300_4f5c_b532_2d1b4d9e3ef4.slice/crio-fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c WatchSource:0}: Error finding container fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c: Status 404 returned error can't find the container with id fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c Nov 01 06:15:36.530007 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:36.529968 4425 generic.go:356] "Generic (PLEG): container finished" podID="39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" containerID="c8c0bf6d234f13ec81562ccab89476fdfadb23ff140b2ec2b7de4a98f39bd3a5" exitCode=0 Nov 01 06:15:36.530429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:36.530147 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" event={"ID":"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4","Type":"ContainerDied","Data":"c8c0bf6d234f13ec81562ccab89476fdfadb23ff140b2ec2b7de4a98f39bd3a5"} Nov 01 06:15:36.530429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:36.530197 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" event={"ID":"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4","Type":"ContainerStarted","Data":"fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c"} Nov 01 06:15:38.035694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.035661 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:38.040485 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.040460 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\") pod \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " Nov 01 06:15:38.040619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.040494 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\" (UniqueName: \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\") pod \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " Nov 01 06:15:38.040619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.040592 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\") pod \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\" (UID: \"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4\") " Nov 01 06:15:38.040959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.040937 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca" pod "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" (UID: "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:15:38.103849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.103811 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca" pod "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" (UID: "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:15:38.109790 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.109759 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t" pod "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" (UID: "39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4"). InnerVolumeSpecName "kube-api-access-tg25t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:38.165339 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.165307 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg25t\" (UniqueName: \"kubernetes.io/projected/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-kube-api-access-tg25t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:38.165575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.165555 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:38.165714 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.165698 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:38.573317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.573277 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" event={"ID":"39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4","Type":"ContainerDied","Data":"fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c"} Nov 01 06:15:38.573653 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.573633 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbbad5bb296cd145b8eff8a70da31d2b46e38a49c3c58ff7035d73fe1e3d7d3c" Nov 01 06:15:38.573857 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:38.573841 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml" Nov 01 06:15:47.553013 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:47.552913 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:15:54.061279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:54.056108 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8"] Nov 01 06:15:54.106279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:54.100767 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9t2g8"] Nov 01 06:15:54.972215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:15:54.972183 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e73879-43b0-475b-86f1-8e8badb0d430" path="/var/lib/kubelet/pods/55e73879-43b0-475b-86f1-8e8badb0d430/volumes" Nov 01 06:15:56.947632 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:15:56.947592 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:16:00.245913 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.245871 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb"] Nov 01 06:16:00.246350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.246292 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:16:00.246350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.246304 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2970c75-769e-4f3d-b61c-06dd0dfaed7c" containerName="cleanup" Nov 01 06:16:00.246350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.246316 4425 cpu_manager.go:414] "containerMap: removing container" podUID="39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" containerName="revoke-keys" Nov 01 06:16:00.246350 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.246321 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" containerName="revoke-keys" Nov 01 06:16:00.246513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.246405 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" containerName="revoke-keys" Nov 01 06:16:00.250095 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.250071 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.303799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.303764 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:16:00.354641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.354603 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:16:00.354916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.354614 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:16:00.354916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.354635 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:16:00.357078 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.357040 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb"] Nov 01 06:16:00.361412 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.361376 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.361557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.361435 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnp8\" (UniqueName: \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.361557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.361544 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.463409 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.463350 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.463642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.463425 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnp8\" (UniqueName: \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.463642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.463473 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.464328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.464295 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.466602 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.466570 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.526081 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.525992 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnp8\" (UniqueName: \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\") pod \"maas-api-revoke-sub-3e31a506-kcpdb\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.565100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.565041 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:00.957884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:00.957848 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb"] Nov 01 06:16:00.959525 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:16:00.959491 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4383ede_e84a_4b52_ba7d_9e3d44ebf91d.slice/crio-b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c WatchSource:0}: Error finding container b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c: Status 404 returned error can't find the container with id b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c Nov 01 06:16:01.740675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:01.740635 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" event={"ID":"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d","Type":"ContainerStarted","Data":"5a8a8309f1b497a217577b44a3fc6b528365c9422a8ddda7042510cd1c87c60f"} Nov 01 06:16:01.741087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:01.740700 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" event={"ID":"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d","Type":"ContainerStarted","Data":"b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c"} Nov 01 06:16:01.897279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:01.884570 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" podStartSLOduration=1.884546442 podStartE2EDuration="1.884546442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:01.86951964 +0000 UTC m=+4377.010108559" watchObservedRunningTime="2026-11-01 06:16:01.884546442 +0000 UTC m=+4377.025135365" Nov 01 06:16:02.578758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.578714 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" podUID="193d32db-a35f-41fe-a727-b5925d24343f" containerName="main" containerID="cri-o://fb4351aa702a2a75beebb3345490d9733ac8fdc048d5035119acaf076be1b20e" gracePeriod=45 Nov 01 06:16:02.754349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.751236 4425 generic.go:356] "Generic (PLEG): container finished" podID="193d32db-a35f-41fe-a727-b5925d24343f" containerID="fb4351aa702a2a75beebb3345490d9733ac8fdc048d5035119acaf076be1b20e" exitCode=0 Nov 01 06:16:02.754349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.751491 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" event={"ID":"193d32db-a35f-41fe-a727-b5925d24343f","Type":"ContainerDied","Data":"fb4351aa702a2a75beebb3345490d9733ac8fdc048d5035119acaf076be1b20e"} Nov 01 06:16:02.754349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.753857 4425 generic.go:356] "Generic (PLEG): container finished" podID="e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" containerID="5a8a8309f1b497a217577b44a3fc6b528365c9422a8ddda7042510cd1c87c60f" exitCode=0 Nov 01 06:16:02.754349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.753907 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" event={"ID":"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d","Type":"ContainerDied","Data":"5a8a8309f1b497a217577b44a3fc6b528365c9422a8ddda7042510cd1c87c60f"} Nov 01 06:16:02.894907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.894875 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:16:02.924205 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924173 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\" (UniqueName: \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924226 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\" (UniqueName: \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924315 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924339 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924388 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924445 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\") pod \"193d32db-a35f-41fe-a727-b5925d24343f\" (UID: \"193d32db-a35f-41fe-a727-b5925d24343f\") " Nov 01 06:16:02.924776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924608 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:16:02.924836 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924817 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:16:02.924911 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.924893 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:02.925120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.925092 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:16:02.926722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.926701 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:16:02.927003 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.926981 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:02.927239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:02.927220 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz" pod "193d32db-a35f-41fe-a727-b5925d24343f" (UID: "193d32db-a35f-41fe-a727-b5925d24343f"). InnerVolumeSpecName "kube-api-access-sgdmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:03.026018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.025971 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.026018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.026001 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgdmz\" (UniqueName: \"kubernetes.io/projected/193d32db-a35f-41fe-a727-b5925d24343f-kube-api-access-sgdmz\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.026018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.026012 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193d32db-a35f-41fe-a727-b5925d24343f-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.026018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.026021 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.026018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.026029 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193d32db-a35f-41fe-a727-b5925d24343f-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.760422 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.760390 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" Nov 01 06:16:03.760422 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.760413 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh" event={"ID":"193d32db-a35f-41fe-a727-b5925d24343f","Type":"ContainerDied","Data":"6b9cbd9b5e6ebb5121bf71b92eef5b04b01abd7325b8219ce489cd32630b711a"} Nov 01 06:16:03.761049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.760469 4425 scope.go:122] "RemoveContainer" containerID="fb4351aa702a2a75beebb3345490d9733ac8fdc048d5035119acaf076be1b20e" Nov 01 06:16:03.899347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.899322 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:03.927488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.927461 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:16:03.936081 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.936057 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\") pod \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " Nov 01 06:16:03.936215 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.936108 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\" (UniqueName: \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\") pod \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " Nov 01 06:16:03.936317 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.936300 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\") pod \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\" (UID: \"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d\") " Nov 01 06:16:03.936509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.936486 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca" pod "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" (UID: "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:16:03.936625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.936609 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:03.938346 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.938327 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8" pod "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" (UID: "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d"). InnerVolumeSpecName "kube-api-access-prnp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:03.938494 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.938474 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca" pod "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" (UID: "e4383ede-e84a-4b52-ba7d-9e3d44ebf91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:03.978487 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:03.978453 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f6cbc132-kserve-5889cd8685-78cbh"] Nov 01 06:16:04.037584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.037508 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:04.037584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.037535 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prnp8\" (UniqueName: \"kubernetes.io/projected/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d-kube-api-access-prnp8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:04.767509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.767472 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" Nov 01 06:16:04.767995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.767475 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb" event={"ID":"e4383ede-e84a-4b52-ba7d-9e3d44ebf91d","Type":"ContainerDied","Data":"b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c"} Nov 01 06:16:04.767995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.767612 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b48a53c808e21abdd7dd382e601355310327d13aa88b303fb4bfb8a17a674c2c" Nov 01 06:16:04.973962 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:04.973928 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193d32db-a35f-41fe-a727-b5925d24343f" path="/var/lib/kubelet/pods/193d32db-a35f-41fe-a727-b5925d24343f/volumes" Nov 01 06:16:08.593053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:08.593021 4425 scope.go:122] "RemoveContainer" containerID="9b21fa052a7f15f19795eb09528aab68fda4523711837a54d98ac3e7e808958a" Nov 01 06:16:16.921768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:16.921722 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:16:16.922354 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:16.921801 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:16:25.421114 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421069 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr"] Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421520 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" containerName="revoke-keys" Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421534 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" containerName="revoke-keys" Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421566 4425 cpu_manager.go:414] "containerMap: removing container" podUID="193d32db-a35f-41fe-a727-b5925d24343f" containerName="main" Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421572 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193d32db-a35f-41fe-a727-b5925d24343f" containerName="main" Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421638 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" containerName="revoke-keys" Nov 01 06:16:25.421669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.421650 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="193d32db-a35f-41fe-a727-b5925d24343f" containerName="main" Nov 01 06:16:25.424867 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.424850 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.443134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.443108 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.443285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.443140 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.443285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.443164 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdt8\" (UniqueName: \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.478995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.478966 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:16:25.529500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.529424 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:16:25.529500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.529443 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:16:25.529791 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.529622 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:16:25.531958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.531931 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr"] Nov 01 06:16:25.543758 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.543737 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.543878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.543765 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.543878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.543797 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdt8\" (UniqueName: \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.544493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.544472 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.546315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.546297 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.606472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.606430 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdt8\" (UniqueName: \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\") pod \"maas-api-revoke-sub-7f76f163-txzwr\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:25.739455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:25.739359 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:26.117170 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:26.117140 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr"] Nov 01 06:16:26.118051 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:16:26.118007 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578b4c49_c10f_4a51_95a8_db5a2ce81e03.slice/crio-c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871 WatchSource:0}: Error finding container c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871: Status 404 returned error can't find the container with id c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871 Nov 01 06:16:26.919100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:26.919063 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" event={"ID":"578b4c49-c10f-4a51-95a8-db5a2ce81e03","Type":"ContainerStarted","Data":"f5210aa573f2e805e22a4ee7817b2383e24999e856cf075aefeeef1af3424b27"} Nov 01 06:16:26.919609 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:26.919121 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" event={"ID":"578b4c49-c10f-4a51-95a8-db5a2ce81e03","Type":"ContainerStarted","Data":"c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871"} Nov 01 06:16:26.949686 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:16:26.949655 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:16:27.037860 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:27.037801 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" podStartSLOduration=2.03778905 podStartE2EDuration="2.03778905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:27.03623039 +0000 UTC m=+4402.176819310" watchObservedRunningTime="2026-11-01 06:16:27.03778905 +0000 UTC m=+4402.178377968" Nov 01 06:16:27.925798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:27.925762 4425 generic.go:356] "Generic (PLEG): container finished" podID="578b4c49-c10f-4a51-95a8-db5a2ce81e03" containerID="f5210aa573f2e805e22a4ee7817b2383e24999e856cf075aefeeef1af3424b27" exitCode=0 Nov 01 06:16:27.926207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:27.925848 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" event={"ID":"578b4c49-c10f-4a51-95a8-db5a2ce81e03","Type":"ContainerDied","Data":"f5210aa573f2e805e22a4ee7817b2383e24999e856cf075aefeeef1af3424b27"} Nov 01 06:16:29.074178 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.074148 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:29.174181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.174150 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\" (UniqueName: \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\") pod \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " Nov 01 06:16:29.174416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.174210 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\") pod \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " Nov 01 06:16:29.174416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.174271 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\") pod \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\" (UID: \"578b4c49-c10f-4a51-95a8-db5a2ce81e03\") " Nov 01 06:16:29.174625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.174586 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca" pod "578b4c49-c10f-4a51-95a8-db5a2ce81e03" (UID: "578b4c49-c10f-4a51-95a8-db5a2ce81e03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:16:29.176538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.176514 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca" pod "578b4c49-c10f-4a51-95a8-db5a2ce81e03" (UID: "578b4c49-c10f-4a51-95a8-db5a2ce81e03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:29.176644 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.176631 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8" pod "578b4c49-c10f-4a51-95a8-db5a2ce81e03" (UID: "578b4c49-c10f-4a51-95a8-db5a2ce81e03"). InnerVolumeSpecName "kube-api-access-phdt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:29.276041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.276009 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdt8\" (UniqueName: \"kubernetes.io/projected/578b4c49-c10f-4a51-95a8-db5a2ce81e03-kube-api-access-phdt8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:29.276041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.276036 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:29.276041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.276045 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578b4c49-c10f-4a51-95a8-db5a2ce81e03-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:29.937473 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.937442 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" Nov 01 06:16:29.937652 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.937470 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr" event={"ID":"578b4c49-c10f-4a51-95a8-db5a2ce81e03","Type":"ContainerDied","Data":"c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871"} Nov 01 06:16:29.937652 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:29.937542 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c341a3a74cc8824766f17ad03990fe125a0d22936e299884e2fd47d43a1a1871" Nov 01 06:16:42.161494 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.161462 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:16:42.161922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.161891 4425 cpu_manager.go:414] "containerMap: removing container" podUID="578b4c49-c10f-4a51-95a8-db5a2ce81e03" containerName="revoke-keys" Nov 01 06:16:42.161922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.161903 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578b4c49-c10f-4a51-95a8-db5a2ce81e03" containerName="revoke-keys" Nov 01 06:16:42.162001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.161960 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="578b4c49-c10f-4a51-95a8-db5a2ce81e03" containerName="revoke-keys" Nov 01 06:16:42.165347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.165331 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.230371 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.230335 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:16:42.318196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.318159 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fqn\" (UniqueName: \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.318424 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.318390 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.419960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.419868 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.419960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.419934 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fqn\" (UniqueName: \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.420636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.420615 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.482591 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.482556 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fqn\" (UniqueName: \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\") pod \"payload-pre-processing-5fb6d674c6-8c4kb\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:42.778108 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:42.778008 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:43.155159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:43.155129 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:16:43.156367 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:16:43.156332 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748bcc3_4454_48b8_9526_d4790374400f.slice/crio-7efd8b23bd6127ea89fcc0dac1745bb1e563c930283ca4c3258adc0a926dcde4 WatchSource:0}: Error finding container 7efd8b23bd6127ea89fcc0dac1745bb1e563c930283ca4c3258adc0a926dcde4: Status 404 returned error can't find the container with id 7efd8b23bd6127ea89fcc0dac1745bb1e563c930283ca4c3258adc0a926dcde4 Nov 01 06:16:44.009946 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:44.009906 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" event={"ID":"4748bcc3-4454-48b8-9526-d4790374400f","Type":"ContainerStarted","Data":"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325"} Nov 01 06:16:44.010404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:44.009972 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" event={"ID":"4748bcc3-4454-48b8-9526-d4790374400f","Type":"ContainerStarted","Data":"7efd8b23bd6127ea89fcc0dac1745bb1e563c930283ca4c3258adc0a926dcde4"} Nov 01 06:16:44.010404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:44.010004 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:44.130031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:44.129975 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" podStartSLOduration=2.129961351 podStartE2EDuration="2.129961351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:44.127846925 +0000 UTC m=+4419.268435862" watchObservedRunningTime="2026-11-01 06:16:44.129961351 +0000 UTC m=+4419.270550270" Nov 01 06:16:49.046968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:49.046933 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw"] Nov 01 06:16:49.097848 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:49.097817 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hskhw"] Nov 01 06:16:50.972626 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:50.972592 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2be570-4d25-4115-bd3e-50cb9f40dc1d" path="/var/lib/kubelet/pods/0f2be570-4d25-4115-bd3e-50cb9f40dc1d/volumes" Nov 01 06:16:55.017608 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:55.017580 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:16:55.188709 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:55.188675 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 06:16:56.947008 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:16:56.946963 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:16:57.130468 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.130438 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:16:57.134437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.134420 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.191429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.191396 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:16:57.264617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.264522 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzk2\" (UniqueName: \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.264804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.264666 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.365913 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.365875 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.366104 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.365960 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzk2\" (UniqueName: \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.366592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.366570 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.428565 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.428528 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzk2\" (UniqueName: \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\") pod \"payload-processing-6dbdbcbf6f-6dq5z\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.447372 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.447338 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:57.825941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:57.825916 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:16:57.827348 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:16:57.827318 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd037175d_ea7b_4622_a1d9_aff1d2ead403.slice/crio-e33be53d01a9807af1d11cda046416714114106d06a94170b4d91e2ff197ad92 WatchSource:0}: Error finding container e33be53d01a9807af1d11cda046416714114106d06a94170b4d91e2ff197ad92: Status 404 returned error can't find the container with id e33be53d01a9807af1d11cda046416714114106d06a94170b4d91e2ff197ad92 Nov 01 06:16:58.079680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:58.079580 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" event={"ID":"d037175d-ea7b-4622-a1d9-aff1d2ead403","Type":"ContainerStarted","Data":"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999"} Nov 01 06:16:58.079680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:58.079644 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" event={"ID":"d037175d-ea7b-4622-a1d9-aff1d2ead403","Type":"ContainerStarted","Data":"e33be53d01a9807af1d11cda046416714114106d06a94170b4d91e2ff197ad92"} Nov 01 06:16:58.080146 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:58.079794 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:16:58.199053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:16:58.198997 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" podStartSLOduration=1.198980783 podStartE2EDuration="1.198980783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:58.197181913 +0000 UTC m=+4433.337770847" watchObservedRunningTime="2026-11-01 06:16:58.198980783 +0000 UTC m=+4433.339569704" Nov 01 06:17:00.189532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.189468 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" podUID="b32e5869-ff99-49e7-ad8e-e032c4d44356" containerName="payload-pre-processing" containerID="cri-o://3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10" gracePeriod=25 Nov 01 06:17:00.443404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.443340 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 06:17:00.598141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.598099 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\") pod \"b32e5869-ff99-49e7-ad8e-e032c4d44356\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " Nov 01 06:17:00.598141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.598131 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\" (UniqueName: \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\") pod \"b32e5869-ff99-49e7-ad8e-e032c4d44356\" (UID: \"b32e5869-ff99-49e7-ad8e-e032c4d44356\") " Nov 01 06:17:00.598552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.598528 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume" pod "b32e5869-ff99-49e7-ad8e-e032c4d44356" (UID: "b32e5869-ff99-49e7-ad8e-e032c4d44356"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:17:00.600594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.600566 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9" pod "b32e5869-ff99-49e7-ad8e-e032c4d44356" (UID: "b32e5869-ff99-49e7-ad8e-e032c4d44356"). InnerVolumeSpecName "kube-api-access-vv2m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:17:00.699161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.699070 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32e5869-ff99-49e7-ad8e-e032c4d44356-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:00.699161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:00.699100 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv2m9\" (UniqueName: \"kubernetes.io/projected/b32e5869-ff99-49e7-ad8e-e032c4d44356-kube-api-access-vv2m9\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:01.096456 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.096425 4425 generic.go:356] "Generic (PLEG): container finished" podID="b32e5869-ff99-49e7-ad8e-e032c4d44356" containerID="3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10" exitCode=0 Nov 01 06:17:01.096618 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.096538 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" Nov 01 06:17:01.096618 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.096559 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" event={"ID":"b32e5869-ff99-49e7-ad8e-e032c4d44356","Type":"ContainerDied","Data":"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10"} Nov 01 06:17:01.096722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.096623 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j678p" event={"ID":"b32e5869-ff99-49e7-ad8e-e032c4d44356","Type":"ContainerDied","Data":"f1941e749c3ae85d22abd17687c2dd5ce89fd344a16f11323ecfbaeb2ce33380"} Nov 01 06:17:01.096722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.096656 4425 scope.go:122] "RemoveContainer" containerID="3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10" Nov 01 06:17:01.105778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.105758 4425 scope.go:122] "RemoveContainer" containerID="3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10" Nov 01 06:17:01.106077 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:01.106058 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10\": container with ID starting with 3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10 not found: ID does not exist" containerID="3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10" Nov 01 06:17:01.106158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.106090 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10"} err="failed to get container status \"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10\": rpc error: code = NotFound desc = could not find container \"3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10\": container with ID starting with 3fc3f01e60a5b3e00138b5a0eae93b632f49c2df06bc76d8c3158f43a0d8ee10 not found: ID does not exist" Nov 01 06:17:01.264341 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.264241 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 06:17:01.268155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:01.268122 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j678p"] Nov 01 06:17:02.972170 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:02.972134 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32e5869-ff99-49e7-ad8e-e032c4d44356" path="/var/lib/kubelet/pods/b32e5869-ff99-49e7-ad8e-e032c4d44356/volumes" Nov 01 06:17:08.625098 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:08.625059 4425 scope.go:122] "RemoveContainer" containerID="48dc0a2ce913f161b48ece0f2ee7adbf5aceebdedf9fbb7db8b488b7f4d44bd1" Nov 01 06:17:09.087833 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:09.087802 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:17:09.258576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:09.258539 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 06:17:11.420881 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.420845 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-b6ff555c6-lxd69"] Nov 01 06:17:11.421355 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.421333 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b32e5869-ff99-49e7-ad8e-e032c4d44356" containerName="payload-pre-processing" Nov 01 06:17:11.421355 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.421347 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32e5869-ff99-49e7-ad8e-e032c4d44356" containerName="payload-pre-processing" Nov 01 06:17:11.421451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.421408 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32e5869-ff99-49e7-ad8e-e032c4d44356" containerName="payload-pre-processing" Nov 01 06:17:11.425978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.425959 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.481979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.481949 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b6ff555c6-lxd69"] Nov 01 06:17:11.492855 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.492826 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvgq\" (UniqueName: \"kubernetes.io/projected/c798be11-e998-4882-9ce6-6806a72664a2-kube-api-access-9cvgq\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.493004 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.492862 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c798be11-e998-4882-9ce6-6806a72664a2-plugins-config-volume\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.585044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.585001 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg"] Nov 01 06:17:11.590662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.590634 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.593419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.593396 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvgq\" (UniqueName: \"kubernetes.io/projected/c798be11-e998-4882-9ce6-6806a72664a2-kube-api-access-9cvgq\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.593573 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.593433 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c798be11-e998-4882-9ce6-6806a72664a2-plugins-config-volume\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.594143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.594124 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c798be11-e998-4882-9ce6-6806a72664a2-plugins-config-volume\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.646967 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.646926 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg"] Nov 01 06:17:11.656223 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.656192 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvgq\" (UniqueName: \"kubernetes.io/projected/c798be11-e998-4882-9ce6-6806a72664a2-kube-api-access-9cvgq\") pod \"payload-processing-b6ff555c6-lxd69\" (UID: \"c798be11-e998-4882-9ce6-6806a72664a2\") " pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.694753 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.694663 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564kn\" (UniqueName: \"kubernetes.io/projected/fbb1ee25-949c-4971-aae8-944e3618f16d-kube-api-access-564kn\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.694901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.694815 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1ee25-949c-4971-aae8-944e3618f16d-plugins-config-volume\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.739097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.739054 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:11.796090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.796052 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1ee25-949c-4971-aae8-944e3618f16d-plugins-config-volume\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.796240 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.796163 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564kn\" (UniqueName: \"kubernetes.io/projected/fbb1ee25-949c-4971-aae8-944e3618f16d-kube-api-access-564kn\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.797513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.797242 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1ee25-949c-4971-aae8-944e3618f16d-plugins-config-volume\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.859031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.858989 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564kn\" (UniqueName: \"kubernetes.io/projected/fbb1ee25-949c-4971-aae8-944e3618f16d-kube-api-access-564kn\") pod \"payload-pre-processing-55775b4dc9-fvzqg\" (UID: \"fbb1ee25-949c-4971-aae8-944e3618f16d\") " pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:11.904871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:11.904837 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:12.116997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:12.116967 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b6ff555c6-lxd69"] Nov 01 06:17:12.117797 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:17:12.117772 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798be11_e998_4882_9ce6_6806a72664a2.slice/crio-b94eb344b12325cbbff7ab8abddc9c925e223aa5d8cf95d59ad5300743619cc0 WatchSource:0}: Error finding container b94eb344b12325cbbff7ab8abddc9c925e223aa5d8cf95d59ad5300743619cc0: Status 404 returned error can't find the container with id b94eb344b12325cbbff7ab8abddc9c925e223aa5d8cf95d59ad5300743619cc0 Nov 01 06:17:12.161455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:12.161423 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" event={"ID":"c798be11-e998-4882-9ce6-6806a72664a2","Type":"ContainerStarted","Data":"b94eb344b12325cbbff7ab8abddc9c925e223aa5d8cf95d59ad5300743619cc0"} Nov 01 06:17:12.281193 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:12.281162 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg"] Nov 01 06:17:12.282446 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:17:12.282419 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb1ee25_949c_4971_aae8_944e3618f16d.slice/crio-80cef1a318fd90946d80af2cec33577cd868b535c6a4ef8571ef1262efb29982 WatchSource:0}: Error finding container 80cef1a318fd90946d80af2cec33577cd868b535c6a4ef8571ef1262efb29982: Status 404 returned error can't find the container with id 80cef1a318fd90946d80af2cec33577cd868b535c6a4ef8571ef1262efb29982 Nov 01 06:17:13.168775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.168730 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" event={"ID":"c798be11-e998-4882-9ce6-6806a72664a2","Type":"ContainerStarted","Data":"c5198858a0b0f3ee0bedec60d3b35c1906f7e896ecb44ca6a459c428b88cf2e0"} Nov 01 06:17:13.169310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.168806 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:13.170351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.170327 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" event={"ID":"fbb1ee25-949c-4971-aae8-944e3618f16d","Type":"ContainerStarted","Data":"e2500dfe648f017d8acbed6613a486183f9ccd45c2596952839dcc197e961f50"} Nov 01 06:17:13.170472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.170375 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" event={"ID":"fbb1ee25-949c-4971-aae8-944e3618f16d","Type":"ContainerStarted","Data":"80cef1a318fd90946d80af2cec33577cd868b535c6a4ef8571ef1262efb29982"} Nov 01 06:17:13.170472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.170411 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:13.287379 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.287327 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" podStartSLOduration=2.287312878 podStartE2EDuration="2.287312878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:13.286283016 +0000 UTC m=+4448.426871936" watchObservedRunningTime="2026-11-01 06:17:13.287312878 +0000 UTC m=+4448.427901796" Nov 01 06:17:13.405327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:13.405265 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" podStartSLOduration=2.405235036 podStartE2EDuration="2.405235036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:13.403844926 +0000 UTC m=+4448.544433847" watchObservedRunningTime="2026-11-01 06:17:13.405235036 +0000 UTC m=+4448.545823955" Nov 01 06:17:14.259069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.259028 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" podUID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" containerName="payload-processing" containerID="cri-o://8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45" gracePeriod=25 Nov 01 06:17:14.504104 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.504074 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 06:17:14.622413 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.622379 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\" (UniqueName: \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\") pod \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " Nov 01 06:17:14.622627 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.622458 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\") pod \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\" (UID: \"ecdd9883-e2d7-48e4-b283-f6f3137e7065\") " Nov 01 06:17:14.624127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.623159 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume" pod "ecdd9883-e2d7-48e4-b283-f6f3137e7065" (UID: "ecdd9883-e2d7-48e4-b283-f6f3137e7065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:17:14.631571 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.631542 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk" pod "ecdd9883-e2d7-48e4-b283-f6f3137e7065" (UID: "ecdd9883-e2d7-48e4-b283-f6f3137e7065"). InnerVolumeSpecName "kube-api-access-rv5kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:17:14.723673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.723630 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5kk\" (UniqueName: \"kubernetes.io/projected/ecdd9883-e2d7-48e4-b283-f6f3137e7065-kube-api-access-rv5kk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:14.723673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:14.723660 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecdd9883-e2d7-48e4-b283-f6f3137e7065-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:15.181958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.181919 4425 generic.go:356] "Generic (PLEG): container finished" podID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" containerID="8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45" exitCode=0 Nov 01 06:17:15.182140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.181964 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" event={"ID":"ecdd9883-e2d7-48e4-b283-f6f3137e7065","Type":"ContainerDied","Data":"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45"} Nov 01 06:17:15.182140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.181968 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" Nov 01 06:17:15.182140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.182014 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rfb94" event={"ID":"ecdd9883-e2d7-48e4-b283-f6f3137e7065","Type":"ContainerDied","Data":"bc0e5c891c4619296e582cf8eb7ad33e9e40f67eaf2dbf0e8ac3633336373f15"} Nov 01 06:17:15.182140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.182037 4425 scope.go:122] "RemoveContainer" containerID="8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45" Nov 01 06:17:15.191407 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.191386 4425 scope.go:122] "RemoveContainer" containerID="8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45" Nov 01 06:17:15.191682 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:15.191663 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45\": container with ID starting with 8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45 not found: ID does not exist" containerID="8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45" Nov 01 06:17:15.191732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.191691 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45"} err="failed to get container status \"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45\": rpc error: code = NotFound desc = could not find container \"8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45\": container with ID starting with 8eaebfcb293097f24ff035f3c75b9b7c8fd881fd61a57e5962d54b2687ef1f45 not found: ID does not exist" Nov 01 06:17:15.349772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.349743 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 06:17:15.400818 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:15.400781 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rfb94"] Nov 01 06:17:16.973434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:16.973350 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" path="/var/lib/kubelet/pods/ecdd9883-e2d7-48e4-b283-f6f3137e7065/volumes" Nov 01 06:17:23.047471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:23.047436 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45"] Nov 01 06:17:23.097918 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:23.097886 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2kr45"] Nov 01 06:17:24.179540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:24.179508 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-55775b4dc9-fvzqg" Nov 01 06:17:24.179930 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:24.179588 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-b6ff555c6-lxd69" Nov 01 06:17:24.350378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:24.350339 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:17:24.444441 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:24.444353 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:17:24.973021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:24.972988 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fde580-8ae9-4942-b76d-ef3a6cfde368" path="/var/lib/kubelet/pods/53fde580-8ae9-4942-b76d-ef3a6cfde368/volumes" Nov 01 06:17:26.947490 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:26.947450 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:17:29.351636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.351595 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" podUID="d037175d-ea7b-4622-a1d9-aff1d2ead403" containerName="payload-processing" containerID="cri-o://0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999" gracePeriod=25 Nov 01 06:17:29.445023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.444984 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" podUID="4748bcc3-4454-48b8-9526-d4790374400f" containerName="payload-pre-processing" containerID="cri-o://4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325" gracePeriod=25 Nov 01 06:17:29.691229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.691201 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:17:29.734520 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.734495 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:17:29.777519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.777485 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\" (UniqueName: \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\") pod \"d037175d-ea7b-4622-a1d9-aff1d2ead403\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " Nov 01 06:17:29.777710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.777578 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\") pod \"d037175d-ea7b-4622-a1d9-aff1d2ead403\" (UID: \"d037175d-ea7b-4622-a1d9-aff1d2ead403\") " Nov 01 06:17:29.777941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.777918 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume" pod "d037175d-ea7b-4622-a1d9-aff1d2ead403" (UID: "d037175d-ea7b-4622-a1d9-aff1d2ead403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:17:29.779965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.779942 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2" pod "d037175d-ea7b-4622-a1d9-aff1d2ead403" (UID: "d037175d-ea7b-4622-a1d9-aff1d2ead403"). InnerVolumeSpecName "kube-api-access-5qzk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:17:29.878751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.878651 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\") pod \"4748bcc3-4454-48b8-9526-d4790374400f\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " Nov 01 06:17:29.878751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.878695 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\" (UniqueName: \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\") pod \"4748bcc3-4454-48b8-9526-d4790374400f\" (UID: \"4748bcc3-4454-48b8-9526-d4790374400f\") " Nov 01 06:17:29.878976 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.878952 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzk2\" (UniqueName: \"kubernetes.io/projected/d037175d-ea7b-4622-a1d9-aff1d2ead403-kube-api-access-5qzk2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:29.878976 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.878963 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d037175d-ea7b-4622-a1d9-aff1d2ead403-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:29.879079 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.879056 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume" pod "4748bcc3-4454-48b8-9526-d4790374400f" (UID: "4748bcc3-4454-48b8-9526-d4790374400f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:17:29.881010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.880986 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn" pod "4748bcc3-4454-48b8-9526-d4790374400f" (UID: "4748bcc3-4454-48b8-9526-d4790374400f"). InnerVolumeSpecName "kube-api-access-m9fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:17:29.979879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.979843 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4748bcc3-4454-48b8-9526-d4790374400f-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:29.979879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:29.979873 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fqn\" (UniqueName: \"kubernetes.io/projected/4748bcc3-4454-48b8-9526-d4790374400f-kube-api-access-m9fqn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:30.261017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.260934 4425 generic.go:356] "Generic (PLEG): container finished" podID="d037175d-ea7b-4622-a1d9-aff1d2ead403" containerID="0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999" exitCode=0 Nov 01 06:17:30.261017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.260995 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" Nov 01 06:17:30.261017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.261010 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" event={"ID":"d037175d-ea7b-4622-a1d9-aff1d2ead403","Type":"ContainerDied","Data":"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999"} Nov 01 06:17:30.261369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.261098 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z" event={"ID":"d037175d-ea7b-4622-a1d9-aff1d2ead403","Type":"ContainerDied","Data":"e33be53d01a9807af1d11cda046416714114106d06a94170b4d91e2ff197ad92"} Nov 01 06:17:30.261369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.261138 4425 scope.go:122] "RemoveContainer" containerID="0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999" Nov 01 06:17:30.262458 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.262439 4425 generic.go:356] "Generic (PLEG): container finished" podID="4748bcc3-4454-48b8-9526-d4790374400f" containerID="4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325" exitCode=0 Nov 01 06:17:30.262579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.262474 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" event={"ID":"4748bcc3-4454-48b8-9526-d4790374400f","Type":"ContainerDied","Data":"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325"} Nov 01 06:17:30.262579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.262503 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" event={"ID":"4748bcc3-4454-48b8-9526-d4790374400f","Type":"ContainerDied","Data":"7efd8b23bd6127ea89fcc0dac1745bb1e563c930283ca4c3258adc0a926dcde4"} Nov 01 06:17:30.262579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.262505 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb" Nov 01 06:17:30.271940 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.271913 4425 scope.go:122] "RemoveContainer" containerID="0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999" Nov 01 06:17:30.272209 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:30.272189 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999\": container with ID starting with 0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999 not found: ID does not exist" containerID="0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999" Nov 01 06:17:30.272321 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.272220 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999"} err="failed to get container status \"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999\": rpc error: code = NotFound desc = could not find container \"0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999\": container with ID starting with 0ec95e313d45705eee3ff304bd5d73fa34d4c26d0ce564b0fb8879d8fd7c5999 not found: ID does not exist" Nov 01 06:17:30.272321 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.272240 4425 scope.go:122] "RemoveContainer" containerID="4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325" Nov 01 06:17:30.281823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.281244 4425 scope.go:122] "RemoveContainer" containerID="4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325" Nov 01 06:17:30.282154 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:30.282133 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325\": container with ID starting with 4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325 not found: ID does not exist" containerID="4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325" Nov 01 06:17:30.282223 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.282165 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325"} err="failed to get container status \"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325\": rpc error: code = NotFound desc = could not find container \"4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325\": container with ID starting with 4c259e42d906789f819d89e9dc1fc13da0011289032ebddc43734e67fdc0c325 not found: ID does not exist" Nov 01 06:17:30.428302 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.428270 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:17:30.431997 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.431971 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5fb6d674c6-8c4kb"] Nov 01 06:17:30.542966 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.542887 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:17:30.546713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.546689 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-6dbdbcbf6f-6dq5z"] Nov 01 06:17:30.972384 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.972354 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4748bcc3-4454-48b8-9526-d4790374400f" path="/var/lib/kubelet/pods/4748bcc3-4454-48b8-9526-d4790374400f/volumes" Nov 01 06:17:30.972753 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:30.972740 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d037175d-ea7b-4622-a1d9-aff1d2ead403" path="/var/lib/kubelet/pods/d037175d-ea7b-4622-a1d9-aff1d2ead403/volumes" Nov 01 06:17:31.394488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394451 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:17:31.394875 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394862 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" containerName="payload-processing" Nov 01 06:17:31.394921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394877 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" containerName="payload-processing" Nov 01 06:17:31.394921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394890 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d037175d-ea7b-4622-a1d9-aff1d2ead403" containerName="payload-processing" Nov 01 06:17:31.394921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394895 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d037175d-ea7b-4622-a1d9-aff1d2ead403" containerName="payload-processing" Nov 01 06:17:31.395027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394927 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4748bcc3-4454-48b8-9526-d4790374400f" containerName="payload-pre-processing" Nov 01 06:17:31.395027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.394933 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4748bcc3-4454-48b8-9526-d4790374400f" containerName="payload-pre-processing" Nov 01 06:17:31.395027 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.395001 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d037175d-ea7b-4622-a1d9-aff1d2ead403" containerName="payload-processing" Nov 01 06:17:31.395126 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.395031 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4748bcc3-4454-48b8-9526-d4790374400f" containerName="payload-pre-processing" Nov 01 06:17:31.395126 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.395042 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdd9883-e2d7-48e4-b283-f6f3137e7065" containerName="payload-processing" Nov 01 06:17:31.399701 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.399681 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.455802 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.455763 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:17:31.495918 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.495882 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.496093 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.496040 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.496093 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.496086 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.496177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.496115 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52gn\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.504173 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504148 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 01 06:17:31.504382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504361 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bvtqv\"" Nov 01 06:17:31.504471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504367 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 06:17:31.504471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504367 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 06:17:31.504471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504443 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 06:17:31.504471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.504449 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 06:17:31.596776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.596741 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.596776 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.596775 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.597056 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.596797 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52gn\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.597056 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.596849 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.597538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.597515 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.597669 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.597634 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.599624 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.599603 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.659599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.659524 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52gn\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\") pod \"payload-processing-external-model-5bcd7997-knxw6\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:31.713120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:31.713074 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:32.088614 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:32.088590 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:17:32.090684 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:17:32.090650 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2526be7f_463d_41ec_b4c8_81149183c64d.slice/crio-9b34605809af41e2c558b0df9c2d01520bab21264df0024d4afbb969e32014c4 WatchSource:0}: Error finding container 9b34605809af41e2c558b0df9c2d01520bab21264df0024d4afbb969e32014c4: Status 404 returned error can't find the container with id 9b34605809af41e2c558b0df9c2d01520bab21264df0024d4afbb969e32014c4 Nov 01 06:17:32.276143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:32.276105 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" event={"ID":"2526be7f-463d-41ec-b4c8-81149183c64d","Type":"ContainerStarted","Data":"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7"} Nov 01 06:17:32.276374 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:32.276166 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" event={"ID":"2526be7f-463d-41ec-b4c8-81149183c64d","Type":"ContainerStarted","Data":"9b34605809af41e2c558b0df9c2d01520bab21264df0024d4afbb969e32014c4"} Nov 01 06:17:32.276374 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:32.276321 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:32.394767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:32.394668 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" podStartSLOduration=1.394649614 podStartE2EDuration="1.394649614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:32.39407842 +0000 UTC m=+4467.534667362" watchObservedRunningTime="2026-11-01 06:17:32.394649614 +0000 UTC m=+4467.535238526" Nov 01 06:17:42.047058 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:42.047017 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77"] Nov 01 06:17:42.090980 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:42.090927 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h6v77"] Nov 01 06:17:42.971852 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:42.971817 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6298c8-379c-40c0-8873-9f34c713dc3f" path="/var/lib/kubelet/pods/2c6298c8-379c-40c0-8873-9f34c713dc3f/volumes" Nov 01 06:17:43.286676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:43.286605 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:17:54.047380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:54.047336 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm"] Nov 01 06:17:54.097896 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:54.097862 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jpllm"] Nov 01 06:17:54.972973 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:17:54.972939 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eeda806-16da-44c7-868c-122b5e5af7e5" path="/var/lib/kubelet/pods/5eeda806-16da-44c7-868c-122b5e5af7e5/volumes" Nov 01 06:17:56.949690 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:17:56.949654 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:18:08.649155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:18:08.649123 4425 scope.go:122] "RemoveContainer" containerID="65bcd0df089646d8eb1b3160e379d95660c1f11360dc1a5575f5f1c368c7448e" Nov 01 06:18:26.952524 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:18:26.952485 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:18:56.948040 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:18:56.947958 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:19:04.046594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:04.046563 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm"] Nov 01 06:19:04.097197 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:04.097165 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06434dd9-bnxnm"] Nov 01 06:19:04.972723 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:04.972691 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337e172d-73b4-44ef-a890-9a1b88dd5ba6" path="/var/lib/kubelet/pods/337e172d-73b4-44ef-a890-9a1b88dd5ba6/volumes" Nov 01 06:19:08.683794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:08.683761 4425 scope.go:122] "RemoveContainer" containerID="9e7061421bf2db671536c54f97153d00a8e3f5a0601803dd9be5212eed3e6b06" Nov 01 06:19:08.692600 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:08.692579 4425 scope.go:122] "RemoveContainer" containerID="becc46bf6ec77e985c285a73db1185b2d60f45e4f346bb7294db71fd2aa51089" Nov 01 06:19:08.700566 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:08.700542 4425 scope.go:122] "RemoveContainer" containerID="83603f3c17333422bbbf5544cf01553fec3d8e8717e4065f099b424e655d4ad2" Nov 01 06:19:22.859282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:22.859218 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:19:23.048161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:23.048126 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x"] Nov 01 06:19:23.051882 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:23.051850 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e651d27e-99m6x"] Nov 01 06:19:24.972515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:24.972482 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f3a6ca-f5cd-44af-9d98-6f7394031e10" path="/var/lib/kubelet/pods/d7f3a6ca-f5cd-44af-9d98-6f7394031e10/volumes" Nov 01 06:19:27.014495 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:19:27.014457 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:19:27.863364 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:27.863315 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" podUID="2526be7f-463d-41ec-b4c8-81149183c64d" containerName="payload-processing" containerID="cri-o://0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7" gracePeriod=25 Nov 01 06:19:28.155437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.155409 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:19:28.226942 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.226913 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\") pod \"2526be7f-463d-41ec-b4c8-81149183c64d\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " Nov 01 06:19:28.227105 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.226953 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\") pod \"2526be7f-463d-41ec-b4c8-81149183c64d\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " Nov 01 06:19:28.227105 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.226987 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\") pod \"2526be7f-463d-41ec-b4c8-81149183c64d\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " Nov 01 06:19:28.227105 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.227009 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\") pod \"2526be7f-463d-41ec-b4c8-81149183c64d\" (UID: \"2526be7f-463d-41ec-b4c8-81149183c64d\") " Nov 01 06:19:28.227437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.227402 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume" pod "2526be7f-463d-41ec-b4c8-81149183c64d" (UID: "2526be7f-463d-41ec-b4c8-81149183c64d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:19:28.227560 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.227454 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume" pod "2526be7f-463d-41ec-b4c8-81149183c64d" (UID: "2526be7f-463d-41ec-b4c8-81149183c64d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:19:28.229302 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.229279 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn" pod "2526be7f-463d-41ec-b4c8-81149183c64d" (UID: "2526be7f-463d-41ec-b4c8-81149183c64d"). InnerVolumeSpecName "kube-api-access-z52gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:28.229408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.229384 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials" pod "2526be7f-463d-41ec-b4c8-81149183c64d" (UID: "2526be7f-463d-41ec-b4c8-81149183c64d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:28.328546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.328513 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:28.328546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.328538 4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-provider-credentials\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:28.328546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.328548 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z52gn\" (UniqueName: \"kubernetes.io/projected/2526be7f-463d-41ec-b4c8-81149183c64d-kube-api-access-z52gn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:28.328794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.328557 4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2526be7f-463d-41ec-b4c8-81149183c64d-routing-overlay-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:28.921927 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.921891 4425 generic.go:356] "Generic (PLEG): container finished" podID="2526be7f-463d-41ec-b4c8-81149183c64d" containerID="0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7" exitCode=0 Nov 01 06:19:28.922117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.921942 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" Nov 01 06:19:28.922117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.921980 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" event={"ID":"2526be7f-463d-41ec-b4c8-81149183c64d","Type":"ContainerDied","Data":"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7"} Nov 01 06:19:28.922117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.922041 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-knxw6" event={"ID":"2526be7f-463d-41ec-b4c8-81149183c64d","Type":"ContainerDied","Data":"9b34605809af41e2c558b0df9c2d01520bab21264df0024d4afbb969e32014c4"} Nov 01 06:19:28.922117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.922071 4425 scope.go:122] "RemoveContainer" containerID="0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7" Nov 01 06:19:28.932004 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.931987 4425 scope.go:122] "RemoveContainer" containerID="0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7" Nov 01 06:19:28.932323 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:19:28.932300 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7\": container with ID starting with 0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7 not found: ID does not exist" containerID="0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7" Nov 01 06:19:28.932390 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:28.932333 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7"} err="failed to get container status \"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7\": rpc error: code = NotFound desc = could not find container \"0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7\": container with ID starting with 0a441123dc4be06c6bd68a839585ff2acb1e86e1c9f8744efc036bb50091b2a7 not found: ID does not exist" Nov 01 06:19:29.088884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:29.088843 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:19:29.092461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:29.092437 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-knxw6"] Nov 01 06:19:30.971969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:30.971937 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2526be7f-463d-41ec-b4c8-81149183c64d" path="/var/lib/kubelet/pods/2526be7f-463d-41ec-b4c8-81149183c64d/volumes" Nov 01 06:19:41.668754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.668721 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz"] Nov 01 06:19:41.669304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.669137 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2526be7f-463d-41ec-b4c8-81149183c64d" containerName="payload-processing" Nov 01 06:19:41.669304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.669149 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2526be7f-463d-41ec-b4c8-81149183c64d" containerName="payload-processing" Nov 01 06:19:41.669304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.669235 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2526be7f-463d-41ec-b4c8-81149183c64d" containerName="payload-processing" Nov 01 06:19:41.672418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.672400 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.729904 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.729872 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz"] Nov 01 06:19:41.768131 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.768095 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtf8m\" (UniqueName: \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.768131 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.768132 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.768443 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.768152 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.778238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.778200 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:19:41.778238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.778217 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:19:41.778238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.778233 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:19:41.778238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.778206 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:19:41.869369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.869332 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtf8m\" (UniqueName: \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.869369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.869368 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.869647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.869396 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.870056 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.870031 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.871999 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.871977 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.931699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.931625 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtf8m\" (UniqueName: \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\") pod \"maas-api-revoke-sub-c1e271a9-wp9xz\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:41.985858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:41.985820 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:42.358481 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:42.358444 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz"] Nov 01 06:19:42.362329 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:19:42.362296 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22861021_78dc_44f5_aeed_fd4198b260e7.slice/crio-48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be WatchSource:0}: Error finding container 48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be: Status 404 returned error can't find the container with id 48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be Nov 01 06:19:43.009572 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:43.009481 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" event={"ID":"22861021-78dc-44f5-aeed-fd4198b260e7","Type":"ContainerStarted","Data":"b6718153c1a3213f5743217a4b1cdd1f410dcf1785acc0ec991321dd74e2ef90"} Nov 01 06:19:43.009572 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:43.009534 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" event={"ID":"22861021-78dc-44f5-aeed-fd4198b260e7","Type":"ContainerStarted","Data":"48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be"} Nov 01 06:19:43.128824 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:43.128774 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" podStartSLOduration=2.128761583 podStartE2EDuration="2.128761583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:43.128164359 +0000 UTC m=+4598.268753279" watchObservedRunningTime="2026-11-01 06:19:43.128761583 +0000 UTC m=+4598.269350502" Nov 01 06:19:44.015603 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:44.015566 4425 generic.go:356] "Generic (PLEG): container finished" podID="22861021-78dc-44f5-aeed-fd4198b260e7" containerID="b6718153c1a3213f5743217a4b1cdd1f410dcf1785acc0ec991321dd74e2ef90" exitCode=0 Nov 01 06:19:44.016039 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:44.015654 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" event={"ID":"22861021-78dc-44f5-aeed-fd4198b260e7","Type":"ContainerDied","Data":"b6718153c1a3213f5743217a4b1cdd1f410dcf1785acc0ec991321dd74e2ef90"} Nov 01 06:19:45.158207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.158182 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:45.202398 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.202322 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\") pod \"22861021-78dc-44f5-aeed-fd4198b260e7\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " Nov 01 06:19:45.202586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.202515 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\" (UniqueName: \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\") pod \"22861021-78dc-44f5-aeed-fd4198b260e7\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " Nov 01 06:19:45.202586 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.202557 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\") pod \"22861021-78dc-44f5-aeed-fd4198b260e7\" (UID: \"22861021-78dc-44f5-aeed-fd4198b260e7\") " Nov 01 06:19:45.203082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.203043 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca" pod "22861021-78dc-44f5-aeed-fd4198b260e7" (UID: "22861021-78dc-44f5-aeed-fd4198b260e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:19:45.204642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.204610 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca" pod "22861021-78dc-44f5-aeed-fd4198b260e7" (UID: "22861021-78dc-44f5-aeed-fd4198b260e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:19:45.204865 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.204847 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m" pod "22861021-78dc-44f5-aeed-fd4198b260e7" (UID: "22861021-78dc-44f5-aeed-fd4198b260e7"). InnerVolumeSpecName "kube-api-access-xtf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:45.304009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.303912 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtf8m\" (UniqueName: \"kubernetes.io/projected/22861021-78dc-44f5-aeed-fd4198b260e7-kube-api-access-xtf8m\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:45.304009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.303939 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:45.304009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:45.303949 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22861021-78dc-44f5-aeed-fd4198b260e7-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:46.026054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:46.026013 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" event={"ID":"22861021-78dc-44f5-aeed-fd4198b260e7","Type":"ContainerDied","Data":"48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be"} Nov 01 06:19:46.026054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:46.026036 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz" Nov 01 06:19:46.026342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:46.026071 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d336069954f0075ba8875ebc26437b175b4e665ffc494a32270c3e198410be" Nov 01 06:19:54.170592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.170547 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt"] Nov 01 06:19:54.171207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.171049 4425 cpu_manager.go:414] "containerMap: removing container" podUID="22861021-78dc-44f5-aeed-fd4198b260e7" containerName="revoke-keys" Nov 01 06:19:54.171207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.171063 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22861021-78dc-44f5-aeed-fd4198b260e7" containerName="revoke-keys" Nov 01 06:19:54.171207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.171155 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="22861021-78dc-44f5-aeed-fd4198b260e7" containerName="revoke-keys" Nov 01 06:19:54.174394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.174378 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.229083 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.229034 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:19:54.279912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.279878 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:19:54.279912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.279878 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:19:54.279912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.279877 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:19:54.282087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.282053 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt"] Nov 01 06:19:54.292451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.292428 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfc7\" (UniqueName: \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.292513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.292464 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.292656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.292631 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.393569 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.393535 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfc7\" (UniqueName: \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.393756 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.393578 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.393756 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.393668 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.395096 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.395061 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.402944 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.402916 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.456304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.456205 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfc7\" (UniqueName: \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\") pod \"maas-api-revoke-sub-ec843018-bb5pt\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.488077 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.488046 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:54.862235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:54.862208 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt"] Nov 01 06:19:54.864824 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:19:54.864782 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e32eea5_36b3_4a97_9dd9_cebd4941eb09.slice/crio-03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348 WatchSource:0}: Error finding container 03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348: Status 404 returned error can't find the container with id 03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348 Nov 01 06:19:55.073009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:55.072975 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" event={"ID":"9e32eea5-36b3-4a97-9dd9-cebd4941eb09","Type":"ContainerStarted","Data":"03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348"} Nov 01 06:19:56.080113 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:56.080078 4425 generic.go:356] "Generic (PLEG): container finished" podID="9e32eea5-36b3-4a97-9dd9-cebd4941eb09" containerID="b527aff8aac166d0b60468ce1d9d2b411e83c84bd722d0207d06947c0e89fd28" exitCode=0 Nov 01 06:19:56.080582 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:56.080172 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" event={"ID":"9e32eea5-36b3-4a97-9dd9-cebd4941eb09","Type":"ContainerDied","Data":"b527aff8aac166d0b60468ce1d9d2b411e83c84bd722d0207d06947c0e89fd28"} Nov 01 06:19:56.946841 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:19:56.946800 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:19:57.219963 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.219934 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:57.321209 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.321179 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\" (UniqueName: \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\") pod \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " Nov 01 06:19:57.321420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.321282 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\") pod \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " Nov 01 06:19:57.321420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.321329 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\") pod \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\" (UID: \"9e32eea5-36b3-4a97-9dd9-cebd4941eb09\") " Nov 01 06:19:57.321686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.321661 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca" pod "9e32eea5-36b3-4a97-9dd9-cebd4941eb09" (UID: "9e32eea5-36b3-4a97-9dd9-cebd4941eb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:19:57.323750 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.323725 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7" pod "9e32eea5-36b3-4a97-9dd9-cebd4941eb09" (UID: "9e32eea5-36b3-4a97-9dd9-cebd4941eb09"). InnerVolumeSpecName "kube-api-access-bkfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:57.323850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.323754 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca" pod "9e32eea5-36b3-4a97-9dd9-cebd4941eb09" (UID: "9e32eea5-36b3-4a97-9dd9-cebd4941eb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:19:57.422059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.422024 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:57.422059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.422053 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:57.422059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:57.422064 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfc7\" (UniqueName: \"kubernetes.io/projected/9e32eea5-36b3-4a97-9dd9-cebd4941eb09-kube-api-access-bkfc7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:58.094726 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:58.094698 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" Nov 01 06:19:58.094726 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:58.094716 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt" event={"ID":"9e32eea5-36b3-4a97-9dd9-cebd4941eb09","Type":"ContainerDied","Data":"03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348"} Nov 01 06:19:58.094953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:19:58.094767 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ca76a2ae883262b5c2b6924262a0c426518a86fd29ff24549043769ec44348" Nov 01 06:20:08.727347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:08.727316 4425 scope.go:122] "RemoveContainer" containerID="e754184f42de0136bc93798f8b0492a08a574a87b4709ea0cd5271964f58b88e" Nov 01 06:20:26.946580 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:20:26.946539 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:20:29.288562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.288528 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd"] Nov 01 06:20:29.288952 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.288922 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9e32eea5-36b3-4a97-9dd9-cebd4941eb09" containerName="revoke-keys" Nov 01 06:20:29.288952 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.288934 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e32eea5-36b3-4a97-9dd9-cebd4941eb09" containerName="revoke-keys" Nov 01 06:20:29.289032 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.289017 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e32eea5-36b3-4a97-9dd9-cebd4941eb09" containerName="revoke-keys" Nov 01 06:20:29.292300 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.292278 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.347547 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.347490 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:20:29.375989 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.375950 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2pzg\" (UniqueName: \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.376187 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.376168 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.376242 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.376206 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.398334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.398286 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:20:29.398499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.398368 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:20:29.398499 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.398401 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:20:29.400467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.400445 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd"] Nov 01 06:20:29.476808 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.476773 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.476808 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.476808 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.477026 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.476874 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2pzg\" (UniqueName: \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.477564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.477538 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.479408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.479387 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.539100 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.539024 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2pzg\" (UniqueName: \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\") pod \"maas-api-revoke-sub-046e1343-tsdjd\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.604847 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.604817 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:29.978778 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:29.978754 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd"] Nov 01 06:20:29.980984 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:20:29.980950 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58ac426_dd35_4226_b239_ffe0a96dad65.slice/crio-1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42 WatchSource:0}: Error finding container 1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42: Status 404 returned error can't find the container with id 1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42 Nov 01 06:20:30.277416 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:30.277388 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" event={"ID":"f58ac426-dd35-4226-b239-ffe0a96dad65","Type":"ContainerStarted","Data":"202312de94765f02f163ff03b1f57cdbf6afac3bd768803d3dbdc4518660425c"} Nov 01 06:20:30.277571 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:30.277441 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" event={"ID":"f58ac426-dd35-4226-b239-ffe0a96dad65","Type":"ContainerStarted","Data":"1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42"} Nov 01 06:20:30.395230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:30.395173 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" podStartSLOduration=1.395157891 podStartE2EDuration="1.395157891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:30.393863032 +0000 UTC m=+4645.534451971" watchObservedRunningTime="2026-11-01 06:20:30.395157891 +0000 UTC m=+4645.535746807" Nov 01 06:20:31.284442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:31.284356 4425 generic.go:356] "Generic (PLEG): container finished" podID="f58ac426-dd35-4226-b239-ffe0a96dad65" containerID="202312de94765f02f163ff03b1f57cdbf6afac3bd768803d3dbdc4518660425c" exitCode=0 Nov 01 06:20:31.284442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:31.284418 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" event={"ID":"f58ac426-dd35-4226-b239-ffe0a96dad65","Type":"ContainerDied","Data":"202312de94765f02f163ff03b1f57cdbf6afac3bd768803d3dbdc4518660425c"} Nov 01 06:20:32.431872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.431846 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:32.507505 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.507473 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\") pod \"f58ac426-dd35-4226-b239-ffe0a96dad65\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " Nov 01 06:20:32.507679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.507514 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\" (UniqueName: \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\") pod \"f58ac426-dd35-4226-b239-ffe0a96dad65\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " Nov 01 06:20:32.507679 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.507589 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\") pod \"f58ac426-dd35-4226-b239-ffe0a96dad65\" (UID: \"f58ac426-dd35-4226-b239-ffe0a96dad65\") " Nov 01 06:20:32.507902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.507873 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca" pod "f58ac426-dd35-4226-b239-ffe0a96dad65" (UID: "f58ac426-dd35-4226-b239-ffe0a96dad65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:32.509761 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.509738 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg" pod "f58ac426-dd35-4226-b239-ffe0a96dad65" (UID: "f58ac426-dd35-4226-b239-ffe0a96dad65"). InnerVolumeSpecName "kube-api-access-d2pzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:32.509878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.509848 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca" pod "f58ac426-dd35-4226-b239-ffe0a96dad65" (UID: "f58ac426-dd35-4226-b239-ffe0a96dad65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:20:32.608556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.608522 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:32.608556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.608552 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58ac426-dd35-4226-b239-ffe0a96dad65-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:32.608556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:32.608563 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2pzg\" (UniqueName: \"kubernetes.io/projected/f58ac426-dd35-4226-b239-ffe0a96dad65-kube-api-access-d2pzg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:33.297127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:33.297031 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" event={"ID":"f58ac426-dd35-4226-b239-ffe0a96dad65","Type":"ContainerDied","Data":"1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42"} Nov 01 06:20:33.297127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:33.297097 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc2e72899333ea1ffbbc4a3c207ce7199e50472548b8728f55aece544013e42" Nov 01 06:20:33.297127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:33.297038 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd" Nov 01 06:20:39.047675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:39.047643 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml"] Nov 01 06:20:39.098874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:39.098842 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kzhml"] Nov 01 06:20:40.026576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.026538 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm"] Nov 01 06:20:40.026947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.026935 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f58ac426-dd35-4226-b239-ffe0a96dad65" containerName="revoke-keys" Nov 01 06:20:40.026995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.026948 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ac426-dd35-4226-b239-ffe0a96dad65" containerName="revoke-keys" Nov 01 06:20:40.027035 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.027022 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ac426-dd35-4226-b239-ffe0a96dad65" containerName="revoke-keys" Nov 01 06:20:40.030178 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.030162 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.085766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.085738 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:20:40.086319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.085753 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:20:40.086319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.085903 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:20:40.086319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.085914 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:20:40.138513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.138480 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm"] Nov 01 06:20:40.189848 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.189820 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.189848 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.189852 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrwb\" (UniqueName: \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.190099 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.189984 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.290842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.290752 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.290842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.290824 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.291106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.290847 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrwb\" (UniqueName: \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.291601 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.291572 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.293369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.293346 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.352772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.352738 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrwb\" (UniqueName: \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\") pod \"maas-api-revoke-sub-d3fba056-rh7jm\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.642594 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.642564 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:40.973834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:40.973744 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4" path="/var/lib/kubelet/pods/39e0b7c0-d300-4f5c-b532-2d1b4d9e3ef4/volumes" Nov 01 06:20:41.019029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:41.019002 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm"] Nov 01 06:20:41.020637 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:20:41.020593 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3245793a_2524_496a_bb64_1bca4f2d373a.slice/crio-a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368 WatchSource:0}: Error finding container a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368: Status 404 returned error can't find the container with id a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368 Nov 01 06:20:41.337570 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:41.337536 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" event={"ID":"3245793a-2524-496a-bb64-1bca4f2d373a","Type":"ContainerStarted","Data":"a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368"} Nov 01 06:20:42.344028 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:42.343984 4425 generic.go:356] "Generic (PLEG): container finished" podID="3245793a-2524-496a-bb64-1bca4f2d373a" containerID="77100cd27a55e0a1366703ebd58fb65cfdfbbf8fce85e0172dda593c6f62e03c" exitCode=0 Nov 01 06:20:42.344513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:42.344063 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" event={"ID":"3245793a-2524-496a-bb64-1bca4f2d373a","Type":"ContainerDied","Data":"77100cd27a55e0a1366703ebd58fb65cfdfbbf8fce85e0172dda593c6f62e03c"} Nov 01 06:20:43.811279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.811163 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:43.934277 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.934213 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\") pod \"3245793a-2524-496a-bb64-1bca4f2d373a\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " Nov 01 06:20:43.934460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.934307 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\") pod \"3245793a-2524-496a-bb64-1bca4f2d373a\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " Nov 01 06:20:43.934460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.934345 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\" (UniqueName: \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\") pod \"3245793a-2524-496a-bb64-1bca4f2d373a\" (UID: \"3245793a-2524-496a-bb64-1bca4f2d373a\") " Nov 01 06:20:43.935351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.935323 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca" pod "3245793a-2524-496a-bb64-1bca4f2d373a" (UID: "3245793a-2524-496a-bb64-1bca4f2d373a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:20:43.937383 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.937354 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca" pod "3245793a-2524-496a-bb64-1bca4f2d373a" (UID: "3245793a-2524-496a-bb64-1bca4f2d373a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:20:43.939006 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:43.938984 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb" pod "3245793a-2524-496a-bb64-1bca4f2d373a" (UID: "3245793a-2524-496a-bb64-1bca4f2d373a"). InnerVolumeSpecName "kube-api-access-msrwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:20:44.035775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.035668 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:44.035775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.035701 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3245793a-2524-496a-bb64-1bca4f2d373a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:44.035775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.035711 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrwb\" (UniqueName: \"kubernetes.io/projected/3245793a-2524-496a-bb64-1bca4f2d373a-kube-api-access-msrwb\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:20:44.395041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.394996 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" event={"ID":"3245793a-2524-496a-bb64-1bca4f2d373a","Type":"ContainerDied","Data":"a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368"} Nov 01 06:20:44.395376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.395355 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a056e0557a86d7a89c2b97425aca6496246a613867c28f5888161bd0b7b9f368" Nov 01 06:20:44.395589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:20:44.395572 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm" Nov 01 06:20:56.947114 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:20:56.947078 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:21:01.922523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:01.922467 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:21:01.922973 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:01.922570 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:21:04.095856 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:04.095817 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb"] Nov 01 06:21:04.097843 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:04.097814 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-kcpdb"] Nov 01 06:21:04.972371 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:04.972338 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4383ede-e84a-4b52-ba7d-9e3d44ebf91d" path="/var/lib/kubelet/pods/e4383ede-e84a-4b52-ba7d-9e3d44ebf91d/volumes" Nov 01 06:21:08.752347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:08.752320 4425 scope.go:122] "RemoveContainer" containerID="d6dc9c177b91187f8727943cc14dc3564d260bc5812b8af3796c1e65113dddee" Nov 01 06:21:14.666121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.666077 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng"] Nov 01 06:21:14.666561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.666533 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3245793a-2524-496a-bb64-1bca4f2d373a" containerName="revoke-keys" Nov 01 06:21:14.666561 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.666545 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3245793a-2524-496a-bb64-1bca4f2d373a" containerName="revoke-keys" Nov 01 06:21:14.666661 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.666648 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3245793a-2524-496a-bb64-1bca4f2d373a" containerName="revoke-keys" Nov 01 06:21:14.671407 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.671390 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.727130 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.727101 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng"] Nov 01 06:21:14.753472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.753440 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnvq\" (UniqueName: \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.753662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.753494 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.753774 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.753744 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.775318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.775288 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:21:14.775318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.775298 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:21:14.775520 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.775322 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:21:14.776790 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.776775 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:21:14.854950 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.854917 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.855156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.854980 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnvq\" (UniqueName: \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.855156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.855033 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.855901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.855866 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.857595 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.857569 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.917018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.916938 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnvq\" (UniqueName: \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\") pod \"maas-api-revoke-sub-ae594979-ttgng\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:14.984195 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:14.984167 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:15.358046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:15.357998 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng"] Nov 01 06:21:15.359519 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:21:15.359493 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cca638_4642_4e05_9b2d_5fb02a1fa894.slice/crio-db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a WatchSource:0}: Error finding container db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a: Status 404 returned error can't find the container with id db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a Nov 01 06:21:15.583621 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:15.583593 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" event={"ID":"23cca638-4642-4e05-9b2d-5fb02a1fa894","Type":"ContainerStarted","Data":"db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a"} Nov 01 06:21:16.589981 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:16.589939 4425 generic.go:356] "Generic (PLEG): container finished" podID="23cca638-4642-4e05-9b2d-5fb02a1fa894" containerID="ad3febeaa6df9b50da6e615f6736d2fb299d4d413277c39041f3943aab5c06e9" exitCode=0 Nov 01 06:21:16.590481 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:16.590027 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" event={"ID":"23cca638-4642-4e05-9b2d-5fb02a1fa894","Type":"ContainerDied","Data":"ad3febeaa6df9b50da6e615f6736d2fb299d4d413277c39041f3943aab5c06e9"} Nov 01 06:21:17.742696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.742669 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:17.784603 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.784572 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\") pod \"23cca638-4642-4e05-9b2d-5fb02a1fa894\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " Nov 01 06:21:17.784788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.784621 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\" (UniqueName: \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\") pod \"23cca638-4642-4e05-9b2d-5fb02a1fa894\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " Nov 01 06:21:17.784788 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.784762 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\") pod \"23cca638-4642-4e05-9b2d-5fb02a1fa894\" (UID: \"23cca638-4642-4e05-9b2d-5fb02a1fa894\") " Nov 01 06:21:17.785152 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.785126 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca" pod "23cca638-4642-4e05-9b2d-5fb02a1fa894" (UID: "23cca638-4642-4e05-9b2d-5fb02a1fa894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:21:17.786863 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.786842 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca" pod "23cca638-4642-4e05-9b2d-5fb02a1fa894" (UID: "23cca638-4642-4e05-9b2d-5fb02a1fa894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:21:17.787090 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.787066 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq" pod "23cca638-4642-4e05-9b2d-5fb02a1fa894" (UID: "23cca638-4642-4e05-9b2d-5fb02a1fa894"). InnerVolumeSpecName "kube-api-access-xfnvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:17.886234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.886143 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.886234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.886185 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cca638-4642-4e05-9b2d-5fb02a1fa894-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:17.886234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:17.886201 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnvq\" (UniqueName: \"kubernetes.io/projected/23cca638-4642-4e05-9b2d-5fb02a1fa894-kube-api-access-xfnvq\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:18.601888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:18.601852 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" Nov 01 06:21:18.601888 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:18.601876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng" event={"ID":"23cca638-4642-4e05-9b2d-5fb02a1fa894","Type":"ContainerDied","Data":"db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a"} Nov 01 06:21:18.602127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:18.601924 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db497052824bf36baad13ee9fc780cf8942263dd2baa99fb63eb84fbd0dc066a" Nov 01 06:21:26.948227 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:21:26.948191 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:21:30.047794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:30.047759 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr"] Nov 01 06:21:30.098425 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:30.098394 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f76f163-txzwr"] Nov 01 06:21:30.972071 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:30.972038 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578b4c49-c10f-4a51-95a8-db5a2ce81e03" path="/var/lib/kubelet/pods/578b4c49-c10f-4a51-95a8-db5a2ce81e03/volumes" Nov 01 06:21:36.559315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.559280 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4"] Nov 01 06:21:36.559731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.559685 4425 cpu_manager.go:414] "containerMap: removing container" podUID="23cca638-4642-4e05-9b2d-5fb02a1fa894" containerName="revoke-keys" Nov 01 06:21:36.559731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.559697 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cca638-4642-4e05-9b2d-5fb02a1fa894" containerName="revoke-keys" Nov 01 06:21:36.559808 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.559767 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cca638-4642-4e05-9b2d-5fb02a1fa894" containerName="revoke-keys" Nov 01 06:21:36.564068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.564051 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.618395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.618366 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:21:36.620976 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.620948 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4"] Nov 01 06:21:36.668642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.668609 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:21:36.668813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.668647 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:21:36.668813 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.668659 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:21:36.713621 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.713589 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.713796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.713627 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gct8\" (UniqueName: \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.713796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.713653 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.814711 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.814683 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.814986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.814725 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gct8\" (UniqueName: \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.814986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.814749 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.815546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.815520 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.817315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.817293 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.876772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.876735 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gct8\" (UniqueName: \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\") pod \"maas-api-revoke-sub-a4aef7d7-54sl4\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:36.876953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:36.876849 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:37.254913 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:37.254889 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4"] Nov 01 06:21:37.256024 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:21:37.255997 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb7c4f9_857f_487e_8c90_a9268e3f1350.slice/crio-55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c WatchSource:0}: Error finding container 55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c: Status 404 returned error can't find the container with id 55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c Nov 01 06:21:37.729501 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:37.729453 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" event={"ID":"bdb7c4f9-857f-487e-8c90-a9268e3f1350","Type":"ContainerStarted","Data":"0a08d4ef7d436f5ac8c16f9f832642b94d05434f8fe8a424286aec0c9becef27"} Nov 01 06:21:37.730022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:37.729524 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" event={"ID":"bdb7c4f9-857f-487e-8c90-a9268e3f1350","Type":"ContainerStarted","Data":"55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c"} Nov 01 06:21:37.847883 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:37.847833 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" podStartSLOduration=1.84781891 podStartE2EDuration="1.84781891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:37.846150802 +0000 UTC m=+4712.986739722" watchObservedRunningTime="2026-11-01 06:21:37.84781891 +0000 UTC m=+4712.988407885" Nov 01 06:21:38.735736 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:38.735697 4425 generic.go:356] "Generic (PLEG): container finished" podID="bdb7c4f9-857f-487e-8c90-a9268e3f1350" containerID="0a08d4ef7d436f5ac8c16f9f832642b94d05434f8fe8a424286aec0c9becef27" exitCode=0 Nov 01 06:21:38.736174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:38.735783 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" event={"ID":"bdb7c4f9-857f-487e-8c90-a9268e3f1350","Type":"ContainerDied","Data":"0a08d4ef7d436f5ac8c16f9f832642b94d05434f8fe8a424286aec0c9becef27"} Nov 01 06:21:39.896912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:39.896884 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:40.047486 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.047393 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\" (UniqueName: \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\") pod \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " Nov 01 06:21:40.047675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.047506 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\") pod \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " Nov 01 06:21:40.047675 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.047539 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\") pod \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\" (UID: \"bdb7c4f9-857f-487e-8c90-a9268e3f1350\") " Nov 01 06:21:40.047980 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.047954 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca" pod "bdb7c4f9-857f-487e-8c90-a9268e3f1350" (UID: "bdb7c4f9-857f-487e-8c90-a9268e3f1350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:21:40.049790 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.049764 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8" pod "bdb7c4f9-857f-487e-8c90-a9268e3f1350" (UID: "bdb7c4f9-857f-487e-8c90-a9268e3f1350"). InnerVolumeSpecName "kube-api-access-5gct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:21:40.049895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.049786 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca" pod "bdb7c4f9-857f-487e-8c90-a9268e3f1350" (UID: "bdb7c4f9-857f-487e-8c90-a9268e3f1350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:21:40.148405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.148370 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gct8\" (UniqueName: \"kubernetes.io/projected/bdb7c4f9-857f-487e-8c90-a9268e3f1350-kube-api-access-5gct8\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:40.148405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.148401 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:40.148612 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.148417 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb7c4f9-857f-487e-8c90-a9268e3f1350-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:21:40.747685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.747638 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" event={"ID":"bdb7c4f9-857f-487e-8c90-a9268e3f1350","Type":"ContainerDied","Data":"55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c"} Nov 01 06:21:40.747876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.747700 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a849e946dedf80ca6b1813e35f65226eacc2619902bc07810c6bf3067e672c" Nov 01 06:21:40.747876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:21:40.747664 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4" Nov 01 06:21:57.015543 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:21:57.015436 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:22:04.532732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.532696 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh"] Nov 01 06:22:04.533216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.533118 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bdb7c4f9-857f-487e-8c90-a9268e3f1350" containerName="revoke-keys" Nov 01 06:22:04.533216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.533131 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb7c4f9-857f-487e-8c90-a9268e3f1350" containerName="revoke-keys" Nov 01 06:22:04.533216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.533200 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb7c4f9-857f-487e-8c90-a9268e3f1350" containerName="revoke-keys" Nov 01 06:22:04.536359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.536342 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.604174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.604144 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.604373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.604209 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.604435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.604376 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6mp\" (UniqueName: \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.642066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.642024 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:22:04.642066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.642029 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:22:04.642066 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.642024 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:22:04.642359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.642115 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:22:04.644406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.644386 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh"] Nov 01 06:22:04.705408 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.705374 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.705613 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.705436 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6mp\" (UniqueName: \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.705613 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.705497 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.706117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.706084 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.708004 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.707981 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.767809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.767769 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6mp\" (UniqueName: \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\") pod \"maas-api-revoke-sub-79805ca9-p4qbh\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:04.850968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:04.850941 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:05.223327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:05.223298 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh"] Nov 01 06:22:05.225342 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:22:05.225311 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5df34c7_8367_400c_b190_f3f7411b0fc2.slice/crio-b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799 WatchSource:0}: Error finding container b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799: Status 404 returned error can't find the container with id b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799 Nov 01 06:22:05.883075 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:05.883038 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" event={"ID":"c5df34c7-8367-400c-b190-f3f7411b0fc2","Type":"ContainerStarted","Data":"6e0f5d1e01412a7ee057b017d88f63a55f904e3383ac436267af1764c6c6983a"} Nov 01 06:22:05.883624 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:05.883100 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" event={"ID":"c5df34c7-8367-400c-b190-f3f7411b0fc2","Type":"ContainerStarted","Data":"b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799"} Nov 01 06:22:06.001520 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:06.001464 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" podStartSLOduration=2.001448743 podStartE2EDuration="2.001448743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:06.000631418 +0000 UTC m=+4741.141220329" watchObservedRunningTime="2026-11-01 06:22:06.001448743 +0000 UTC m=+4741.142037655" Nov 01 06:22:06.896386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:06.896336 4425 generic.go:356] "Generic (PLEG): container finished" podID="c5df34c7-8367-400c-b190-f3f7411b0fc2" containerID="6e0f5d1e01412a7ee057b017d88f63a55f904e3383ac436267af1764c6c6983a" exitCode=0 Nov 01 06:22:06.896386 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:06.896390 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" event={"ID":"c5df34c7-8367-400c-b190-f3f7411b0fc2","Type":"ContainerDied","Data":"6e0f5d1e01412a7ee057b017d88f63a55f904e3383ac436267af1764c6c6983a"} Nov 01 06:22:08.039992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.039965 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:08.137225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.137189 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\" (UniqueName: \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\") pod \"c5df34c7-8367-400c-b190-f3f7411b0fc2\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " Nov 01 06:22:08.137225 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.137231 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\") pod \"c5df34c7-8367-400c-b190-f3f7411b0fc2\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " Nov 01 06:22:08.137469 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.137287 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\") pod \"c5df34c7-8367-400c-b190-f3f7411b0fc2\" (UID: \"c5df34c7-8367-400c-b190-f3f7411b0fc2\") " Nov 01 06:22:08.137640 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.137615 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca" pod "c5df34c7-8367-400c-b190-f3f7411b0fc2" (UID: "c5df34c7-8367-400c-b190-f3f7411b0fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:22:08.139553 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.139532 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca" pod "c5df34c7-8367-400c-b190-f3f7411b0fc2" (UID: "c5df34c7-8367-400c-b190-f3f7411b0fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:22:08.139635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.139616 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp" pod "c5df34c7-8367-400c-b190-f3f7411b0fc2" (UID: "c5df34c7-8367-400c-b190-f3f7411b0fc2"). InnerVolumeSpecName "kube-api-access-dd6mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:22:08.238674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.238591 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6mp\" (UniqueName: \"kubernetes.io/projected/c5df34c7-8367-400c-b190-f3f7411b0fc2-kube-api-access-dd6mp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:08.238674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.238620 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:08.238674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.238630 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5df34c7-8367-400c-b190-f3f7411b0fc2-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:08.774777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.774745 4425 scope.go:122] "RemoveContainer" containerID="5a8a8309f1b497a217577b44a3fc6b528365c9422a8ddda7042510cd1c87c60f" Nov 01 06:22:08.783713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.783690 4425 scope.go:122] "RemoveContainer" containerID="c8c0bf6d234f13ec81562ccab89476fdfadb23ff140b2ec2b7de4a98f39bd3a5" Nov 01 06:22:08.906850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.906820 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" Nov 01 06:22:08.907046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.906845 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh" event={"ID":"c5df34c7-8367-400c-b190-f3f7411b0fc2","Type":"ContainerDied","Data":"b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799"} Nov 01 06:22:08.907046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:08.906907 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7b4e9d86dc466d97e28e4795eb5d4468391e7d9ab3a8d963083f702768ce799" Nov 01 06:22:21.986280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:21.986152 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2"] Nov 01 06:22:21.997942 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:21.997907 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c5df34c7-8367-400c-b190-f3f7411b0fc2" containerName="revoke-keys" Nov 01 06:22:21.998173 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:21.998158 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5df34c7-8367-400c-b190-f3f7411b0fc2" containerName="revoke-keys" Nov 01 06:22:21.998435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:21.998420 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5df34c7-8367-400c-b190-f3f7411b0fc2" containerName="revoke-keys" Nov 01 06:22:22.012283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.009549 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.026986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.026326 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhgx\" (UniqueName: \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.026986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.026365 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.026986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.026434 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.118278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.117237 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2"] Nov 01 06:22:22.123286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.118627 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:22:22.123286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.118822 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:22:22.123286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.118894 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:22:22.123286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.118943 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:22:22.127289 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.127224 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhgx\" (UniqueName: \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.130965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.130927 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.131136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.131052 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.132685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.132611 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.134548 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.134508 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.195472 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.195446 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhgx\" (UniqueName: \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\") pod \"maas-api-revoke-sub-0b7e8edb-7d9v2\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.336504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.336473 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:22.711976 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:22:22.711939 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc6c8cd_6601_4985_a9d1_69f1e83cb971.slice/crio-ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682 WatchSource:0}: Error finding container ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682: Status 404 returned error can't find the container with id ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682 Nov 01 06:22:22.739236 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:22.739207 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2"] Nov 01 06:22:23.029076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:23.029039 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" event={"ID":"2cc6c8cd-6601-4985-a9d1-69f1e83cb971","Type":"ContainerStarted","Data":"ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682"} Nov 01 06:22:24.035460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:24.035369 4425 generic.go:356] "Generic (PLEG): container finished" podID="2cc6c8cd-6601-4985-a9d1-69f1e83cb971" containerID="0334d28f8e767d30978a36a485bd66ed1c6deb97fd0d9d9b83f990ded603e4b5" exitCode=0 Nov 01 06:22:24.035840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:24.035459 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" event={"ID":"2cc6c8cd-6601-4985-a9d1-69f1e83cb971","Type":"ContainerDied","Data":"0334d28f8e767d30978a36a485bd66ed1c6deb97fd0d9d9b83f990ded603e4b5"} Nov 01 06:22:25.179454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.179424 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:25.262608 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.262573 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\") pod \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " Nov 01 06:22:25.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.262618 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\") pod \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " Nov 01 06:22:25.262780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.262682 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\" (UniqueName: \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\") pod \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\" (UID: \"2cc6c8cd-6601-4985-a9d1-69f1e83cb971\") " Nov 01 06:22:25.262949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.262926 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca" pod "2cc6c8cd-6601-4985-a9d1-69f1e83cb971" (UID: "2cc6c8cd-6601-4985-a9d1-69f1e83cb971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:22:25.264953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.264925 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca" pod "2cc6c8cd-6601-4985-a9d1-69f1e83cb971" (UID: "2cc6c8cd-6601-4985-a9d1-69f1e83cb971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:22:25.265092 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.265075 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx" pod "2cc6c8cd-6601-4985-a9d1-69f1e83cb971" (UID: "2cc6c8cd-6601-4985-a9d1-69f1e83cb971"). InnerVolumeSpecName "kube-api-access-jqhgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:22:25.363668 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.363637 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqhgx\" (UniqueName: \"kubernetes.io/projected/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-kube-api-access-jqhgx\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:25.363668 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.363665 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:25.363668 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:25.363676 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cc6c8cd-6601-4985-a9d1-69f1e83cb971-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:26.046143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:26.046112 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" Nov 01 06:22:26.046324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:26.046113 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2" event={"ID":"2cc6c8cd-6601-4985-a9d1-69f1e83cb971","Type":"ContainerDied","Data":"ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682"} Nov 01 06:22:26.046324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:26.046242 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7eb14fffe54d5c477c3025851b85a0a9cc33e33aaa45e2263dc7d8ef26b682" Nov 01 06:22:26.946558 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:22:26.946521 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:22:33.245727 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.245682 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh"] Nov 01 06:22:33.248231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.246086 4425 cpu_manager.go:414] "containerMap: removing container" podUID="2cc6c8cd-6601-4985-a9d1-69f1e83cb971" containerName="revoke-keys" Nov 01 06:22:33.248231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.246097 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc6c8cd-6601-4985-a9d1-69f1e83cb971" containerName="revoke-keys" Nov 01 06:22:33.248231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.246181 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc6c8cd-6601-4985-a9d1-69f1e83cb971" containerName="revoke-keys" Nov 01 06:22:33.249304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.249278 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.304636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.304600 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:22:33.304636 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.304615 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:22:33.304916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.304677 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:22:33.304916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.304777 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:22:33.307735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.307698 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh"] Nov 01 06:22:33.344502 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.344468 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggfv\" (UniqueName: \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.344706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.344564 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.344706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.344599 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.445190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.445150 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.445424 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.445199 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.445424 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.445245 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggfv\" (UniqueName: \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.445969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.445941 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.447667 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.447646 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.507208 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.507136 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggfv\" (UniqueName: \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\") pod \"maas-api-revoke-sub-3d4156f1-8jpzh\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.563357 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.563318 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:33.939238 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:33.939213 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh"] Nov 01 06:22:33.940976 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:22:33.940951 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788944dc_aaa0_461a_8da5_eb673e708987.slice/crio-a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df WatchSource:0}: Error finding container a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df: Status 404 returned error can't find the container with id a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df Nov 01 06:22:34.086412 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:34.086376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" event={"ID":"788944dc-aaa0-461a-8da5-eb673e708987","Type":"ContainerStarted","Data":"a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df"} Nov 01 06:22:35.092650 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:35.092557 4425 generic.go:356] "Generic (PLEG): container finished" podID="788944dc-aaa0-461a-8da5-eb673e708987" containerID="79c4e09fd8dc407a463e0030229b5a81f460db6667cbc2cf886082dbfa74c865" exitCode=0 Nov 01 06:22:35.092650 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:35.092639 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" event={"ID":"788944dc-aaa0-461a-8da5-eb673e708987","Type":"ContainerDied","Data":"79c4e09fd8dc407a463e0030229b5a81f460db6667cbc2cf886082dbfa74c865"} Nov 01 06:22:36.238816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.238787 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:36.377378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.377296 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\" (UniqueName: \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\") pod \"788944dc-aaa0-461a-8da5-eb673e708987\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " Nov 01 06:22:36.377378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.377364 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\") pod \"788944dc-aaa0-461a-8da5-eb673e708987\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " Nov 01 06:22:36.377592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.377493 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\") pod \"788944dc-aaa0-461a-8da5-eb673e708987\" (UID: \"788944dc-aaa0-461a-8da5-eb673e708987\") " Nov 01 06:22:36.377737 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.377708 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca" pod "788944dc-aaa0-461a-8da5-eb673e708987" (UID: "788944dc-aaa0-461a-8da5-eb673e708987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:22:36.377912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.377834 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:36.379757 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.379734 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv" pod "788944dc-aaa0-461a-8da5-eb673e708987" (UID: "788944dc-aaa0-461a-8da5-eb673e708987"). InnerVolumeSpecName "kube-api-access-hggfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:22:36.379871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.379770 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca" pod "788944dc-aaa0-461a-8da5-eb673e708987" (UID: "788944dc-aaa0-461a-8da5-eb673e708987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:22:36.479118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.479082 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggfv\" (UniqueName: \"kubernetes.io/projected/788944dc-aaa0-461a-8da5-eb673e708987-kube-api-access-hggfv\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:36.479118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:36.479117 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788944dc-aaa0-461a-8da5-eb673e708987-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:22:37.104494 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:37.104459 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" event={"ID":"788944dc-aaa0-461a-8da5-eb673e708987","Type":"ContainerDied","Data":"a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df"} Nov 01 06:22:37.104676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:37.104509 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f40022c16a804d3c55e3096041b4d6cfee67f1a9a44bf4c30558056bc5e5df" Nov 01 06:22:37.104676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:37.104506 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh" Nov 01 06:22:56.947737 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:22:56.947694 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:22:57.373300 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.373266 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh"] Nov 01 06:22:57.423924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.423895 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8jpzh"] Nov 01 06:22:57.493445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.493415 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4"] Nov 01 06:22:57.493971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.493953 4425 cpu_manager.go:414] "containerMap: removing container" podUID="788944dc-aaa0-461a-8da5-eb673e708987" containerName="revoke-keys" Nov 01 06:22:57.494029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.493974 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788944dc-aaa0-461a-8da5-eb673e708987" containerName="revoke-keys" Nov 01 06:22:57.494073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.494065 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="788944dc-aaa0-461a-8da5-eb673e708987" containerName="revoke-keys" Nov 01 06:22:57.497379 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.497357 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.526604 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.526572 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqb8g\" (UniqueName: \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.526735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.526635 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.526735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.526700 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.552429 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.552405 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:22:57.552894 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.552427 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:22:57.552894 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.552556 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:22:57.552894 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.552601 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:22:57.555986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.555964 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4"] Nov 01 06:22:57.627490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.627397 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.627490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.627453 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.627695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.627557 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqb8g\" (UniqueName: \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.628147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.628128 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.630038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.630016 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.689722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.689685 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqb8g\" (UniqueName: \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\") pod \"maas-api-revoke-sub-3d4156f1-9qdc4\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:57.809777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:57.809740 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:22:58.186678 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:58.186606 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4"] Nov 01 06:22:58.188036 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:22:58.188009 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fd68b6_ef29_4e6b_a787_cea377c7bc07.slice/crio-ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0 WatchSource:0}: Error finding container ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0: Status 404 returned error can't find the container with id ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0 Nov 01 06:22:58.237792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:58.237755 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" event={"ID":"76fd68b6-ef29-4e6b-a787-cea377c7bc07","Type":"ContainerStarted","Data":"ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0"} Nov 01 06:22:58.972559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:58.972525 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788944dc-aaa0-461a-8da5-eb673e708987" path="/var/lib/kubelet/pods/788944dc-aaa0-461a-8da5-eb673e708987/volumes" Nov 01 06:22:59.245911 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:59.245817 4425 generic.go:356] "Generic (PLEG): container finished" podID="76fd68b6-ef29-4e6b-a787-cea377c7bc07" containerID="f5a178e5448079f46224b25a5d0092bda914d8bbc11bb029018f85fcaddec07e" exitCode=0 Nov 01 06:22:59.246327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:22:59.245901 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" event={"ID":"76fd68b6-ef29-4e6b-a787-cea377c7bc07","Type":"ContainerDied","Data":"f5a178e5448079f46224b25a5d0092bda914d8bbc11bb029018f85fcaddec07e"} Nov 01 06:23:00.392699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.392665 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:23:00.451405 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.451368 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\") pod \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " Nov 01 06:23:00.451625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.451412 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\") pod \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " Nov 01 06:23:00.451625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.451523 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\" (UniqueName: \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\") pod \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\" (UID: \"76fd68b6-ef29-4e6b-a787-cea377c7bc07\") " Nov 01 06:23:00.451809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.451784 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca" pod "76fd68b6-ef29-4e6b-a787-cea377c7bc07" (UID: "76fd68b6-ef29-4e6b-a787-cea377c7bc07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:23:00.453849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.453821 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g" pod "76fd68b6-ef29-4e6b-a787-cea377c7bc07" (UID: "76fd68b6-ef29-4e6b-a787-cea377c7bc07"). InnerVolumeSpecName "kube-api-access-gqb8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:23:00.453965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.453854 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca" pod "76fd68b6-ef29-4e6b-a787-cea377c7bc07" (UID: "76fd68b6-ef29-4e6b-a787-cea377c7bc07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:23:00.552484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.552403 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:00.552484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.552438 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd68b6-ef29-4e6b-a787-cea377c7bc07-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:00.552484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:00.552448 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqb8g\" (UniqueName: \"kubernetes.io/projected/76fd68b6-ef29-4e6b-a787-cea377c7bc07-kube-api-access-gqb8g\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:01.257396 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:01.257315 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" Nov 01 06:23:01.257396 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:01.257355 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4" event={"ID":"76fd68b6-ef29-4e6b-a787-cea377c7bc07","Type":"ContainerDied","Data":"ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0"} Nov 01 06:23:01.257599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:01.257402 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed19613f730b23cf6ea034be2b56e3805cf388d7cab2d9df1507715ef088cbf0" Nov 01 06:23:08.806637 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:08.806604 4425 scope.go:122] "RemoveContainer" containerID="f5210aa573f2e805e22a4ee7817b2383e24999e856cf075aefeeef1af3424b27" Nov 01 06:23:26.947454 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:23:26.947418 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:23:42.712751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.712711 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65"] Nov 01 06:23:42.713422 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.713357 4425 cpu_manager.go:414] "containerMap: removing container" podUID="76fd68b6-ef29-4e6b-a787-cea377c7bc07" containerName="revoke-keys" Nov 01 06:23:42.713422 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.713378 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fd68b6-ef29-4e6b-a787-cea377c7bc07" containerName="revoke-keys" Nov 01 06:23:42.713567 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.713500 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fd68b6-ef29-4e6b-a787-cea377c7bc07" containerName="revoke-keys" Nov 01 06:23:42.716528 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.716507 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.773872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.773839 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65"] Nov 01 06:23:42.821301 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.821244 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:23:42.821490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.821333 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:23:42.821490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.821340 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:23:42.821490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.821367 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:23:42.845686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.845661 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mt4t\" (UniqueName: \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.845850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.845721 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.845850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.845807 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.946919 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.946883 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mt4t\" (UniqueName: \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.947094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.946952 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.947094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.946977 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.947662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.947641 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:42.949556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:42.949539 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:43.010315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:43.010223 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mt4t\" (UniqueName: \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\") pod \"maas-api-revoke-sub-e2fc9c5b-b5w65\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:43.029144 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:43.029103 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:43.404031 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:43.404007 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65"] Nov 01 06:23:43.405624 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:23:43.405591 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486154bf_ceea_4f30_9803_8f2f0a265b74.slice/crio-c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49 WatchSource:0}: Error finding container c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49: Status 404 returned error can't find the container with id c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49 Nov 01 06:23:43.485783 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:43.485748 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" event={"ID":"486154bf-ceea-4f30-9803-8f2f0a265b74","Type":"ContainerStarted","Data":"c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49"} Nov 01 06:23:44.494562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:44.494522 4425 generic.go:356] "Generic (PLEG): container finished" podID="486154bf-ceea-4f30-9803-8f2f0a265b74" containerID="82a6225f79650148c4a44b66b483d1d8cd2c9a0de21474aebac3f1f4d6b52fae" exitCode=0 Nov 01 06:23:44.495237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:44.495202 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" event={"ID":"486154bf-ceea-4f30-9803-8f2f0a265b74","Type":"ContainerDied","Data":"82a6225f79650148c4a44b66b483d1d8cd2c9a0de21474aebac3f1f4d6b52fae"} Nov 01 06:23:45.641745 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.641712 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:45.775661 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.775580 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\" (UniqueName: \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\") pod \"486154bf-ceea-4f30-9803-8f2f0a265b74\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " Nov 01 06:23:45.775661 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.775622 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\") pod \"486154bf-ceea-4f30-9803-8f2f0a265b74\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " Nov 01 06:23:45.775875 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.775710 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\") pod \"486154bf-ceea-4f30-9803-8f2f0a265b74\" (UID: \"486154bf-ceea-4f30-9803-8f2f0a265b74\") " Nov 01 06:23:45.776112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.776090 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca" pod "486154bf-ceea-4f30-9803-8f2f0a265b74" (UID: "486154bf-ceea-4f30-9803-8f2f0a265b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:23:45.777824 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.777800 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t" pod "486154bf-ceea-4f30-9803-8f2f0a265b74" (UID: "486154bf-ceea-4f30-9803-8f2f0a265b74"). InnerVolumeSpecName "kube-api-access-9mt4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:23:45.777998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.777971 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca" pod "486154bf-ceea-4f30-9803-8f2f0a265b74" (UID: "486154bf-ceea-4f30-9803-8f2f0a265b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:23:45.877497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.877460 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mt4t\" (UniqueName: \"kubernetes.io/projected/486154bf-ceea-4f30-9803-8f2f0a265b74-kube-api-access-9mt4t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:45.877497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.877493 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:45.877497 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:45.877503 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486154bf-ceea-4f30-9803-8f2f0a265b74-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:46.507133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:46.507107 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" Nov 01 06:23:46.507352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:46.507108 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65" event={"ID":"486154bf-ceea-4f30-9803-8f2f0a265b74","Type":"ContainerDied","Data":"c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49"} Nov 01 06:23:46.507352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:46.507239 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31fedce9492f84b1fa27259336fd687625a913758aecd50a8314c47b5a32e49" Nov 01 06:23:53.500323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.500284 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5"] Nov 01 06:23:53.500713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.500697 4425 cpu_manager.go:414] "containerMap: removing container" podUID="486154bf-ceea-4f30-9803-8f2f0a265b74" containerName="revoke-keys" Nov 01 06:23:53.500713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.500710 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486154bf-ceea-4f30-9803-8f2f0a265b74" containerName="revoke-keys" Nov 01 06:23:53.500812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.500789 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="486154bf-ceea-4f30-9803-8f2f0a265b74" containerName="revoke-keys" Nov 01 06:23:53.504063 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.504044 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.562030 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.561998 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5"] Nov 01 06:23:53.609299 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.609265 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:23:53.609299 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.609267 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:23:53.609523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.609304 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:23:53.609523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.609365 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:23:53.651924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.651883 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.651924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.651926 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p7k\" (UniqueName: \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.652136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.652058 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.753683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.753598 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.753860 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.753730 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.753860 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.753767 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p7k\" (UniqueName: \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.754474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.754452 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.756269 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.756228 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.816158 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.816124 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p7k\" (UniqueName: \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\") pod \"maas-api-revoke-sub-5a29efb3-4r4d5\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:53.817793 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:53.817765 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:54.196079 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:54.196053 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5"] Nov 01 06:23:54.198072 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:23:54.198045 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26da6e7a_0b3a_45cd_9869_90cbd83fcd0a.slice/crio-af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a WatchSource:0}: Error finding container af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a: Status 404 returned error can't find the container with id af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a Nov 01 06:23:54.549082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:54.549044 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" event={"ID":"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a","Type":"ContainerStarted","Data":"9f90e3f03ecfd7c46f6e5a8891d0982cf8e55bc82158558950dd0870d3433c95"} Nov 01 06:23:54.549572 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:54.549119 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" event={"ID":"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a","Type":"ContainerStarted","Data":"af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a"} Nov 01 06:23:54.666617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:54.666565 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" podStartSLOduration=1.666549083 podStartE2EDuration="1.666549083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:54.665727595 +0000 UTC m=+4849.806316514" watchObservedRunningTime="2026-11-01 06:23:54.666549083 +0000 UTC m=+4849.807138001" Nov 01 06:23:55.555505 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:55.555474 4425 generic.go:356] "Generic (PLEG): container finished" podID="26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" containerID="9f90e3f03ecfd7c46f6e5a8891d0982cf8e55bc82158558950dd0870d3433c95" exitCode=0 Nov 01 06:23:55.555929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:55.555568 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" event={"ID":"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a","Type":"ContainerDied","Data":"9f90e3f03ecfd7c46f6e5a8891d0982cf8e55bc82158558950dd0870d3433c95"} Nov 01 06:23:56.749056 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.749018 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:23:56.885680 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.885593 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\") pod \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " Nov 01 06:23:56.886059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.886020 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\" (UniqueName: \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\") pod \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " Nov 01 06:23:56.886282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.886233 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\") pod \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\" (UID: \"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a\") " Nov 01 06:23:56.894733 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.894699 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca" pod "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" (UID: "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:23:56.895302 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.895276 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca" pod "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" (UID: "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:23:56.902462 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.902417 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k" pod "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" (UID: "26da6e7a-0b3a-45cd-9869-90cbd83fcd0a"). InnerVolumeSpecName "kube-api-access-p9p7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:23:56.991000 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.990916 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9p7k\" (UniqueName: \"kubernetes.io/projected/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-kube-api-access-p9p7k\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:56.991000 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.990955 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:56.991000 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:56.990972 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:23:57.034277 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:23:57.033285 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:23:57.605278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:57.592593 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" event={"ID":"26da6e7a-0b3a-45cd-9869-90cbd83fcd0a","Type":"ContainerDied","Data":"af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a"} Nov 01 06:23:57.605278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:57.592666 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6d5bbb8e4cad93d8010213e9decbe76e05ee4566891e98ec8fee9703bac12a" Nov 01 06:23:57.605278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:23:57.592745 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5" Nov 01 06:24:26.947725 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:24:26.947686 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:24:39.819774 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.819736 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4"] Nov 01 06:24:39.870410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.870377 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9qdc4"] Nov 01 06:24:39.941566 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.941532 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr"] Nov 01 06:24:39.941945 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.941931 4425 cpu_manager.go:414] "containerMap: removing container" podUID="26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" containerName="revoke-keys" Nov 01 06:24:39.941994 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.941947 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" containerName="revoke-keys" Nov 01 06:24:39.942049 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.942040 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" containerName="revoke-keys" Nov 01 06:24:39.945463 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.945443 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:39.999351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.999285 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:24:39.999351 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.999293 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:24:39.999666 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.999427 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:24:39.999890 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:39.999860 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:24:40.002765 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.002742 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr"] Nov 01 06:24:40.141983 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.141883 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqg4\" (UniqueName: \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.141983 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.141950 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.142234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.142127 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.243582 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.243541 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.243763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.243687 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqg4\" (UniqueName: \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.243763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.243756 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.244388 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.244366 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.246194 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.246170 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.305945 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.305914 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqg4\" (UniqueName: \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\") pod \"maas-api-revoke-sub-3d4156f1-wdlcr\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.558710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.558620 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:40.937018 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.935033 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr"] Nov 01 06:24:40.972823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:40.972792 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fd68b6-ef29-4e6b-a787-cea377c7bc07" path="/var/lib/kubelet/pods/76fd68b6-ef29-4e6b-a787-cea377c7bc07/volumes" Nov 01 06:24:41.824391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:41.824352 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" event={"ID":"4402b7e2-fc19-4538-abdd-a39c587c3f28","Type":"ContainerStarted","Data":"6e747c6501c58fc52165710cb9e589ad8088762bf9f1ceafa1fc0ce23d61686a"} Nov 01 06:24:41.824637 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:41.824407 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" event={"ID":"4402b7e2-fc19-4538-abdd-a39c587c3f28","Type":"ContainerStarted","Data":"fbe0daed1d022001f0835330702a4a60fd0551eb585b5be64977411623fe8374"} Nov 01 06:24:41.943538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:41.943485 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" podStartSLOduration=2.943470059 podStartE2EDuration="2.943470059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:41.941199459 +0000 UTC m=+4897.081788378" watchObservedRunningTime="2026-11-01 06:24:41.943470059 +0000 UTC m=+4897.084059014" Nov 01 06:24:42.831334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:42.831300 4425 generic.go:356] "Generic (PLEG): container finished" podID="4402b7e2-fc19-4538-abdd-a39c587c3f28" containerID="6e747c6501c58fc52165710cb9e589ad8088762bf9f1ceafa1fc0ce23d61686a" exitCode=0 Nov 01 06:24:42.831525 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:42.831391 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" event={"ID":"4402b7e2-fc19-4538-abdd-a39c587c3f28","Type":"ContainerDied","Data":"6e747c6501c58fc52165710cb9e589ad8088762bf9f1ceafa1fc0ce23d61686a"} Nov 01 06:24:43.982418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:43.982386 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:44.076533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.076497 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\") pod \"4402b7e2-fc19-4538-abdd-a39c587c3f28\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " Nov 01 06:24:44.076730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.076585 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\" (UniqueName: \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\") pod \"4402b7e2-fc19-4538-abdd-a39c587c3f28\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " Nov 01 06:24:44.076730 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.076645 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\") pod \"4402b7e2-fc19-4538-abdd-a39c587c3f28\" (UID: \"4402b7e2-fc19-4538-abdd-a39c587c3f28\") " Nov 01 06:24:44.077040 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.077013 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca" pod "4402b7e2-fc19-4538-abdd-a39c587c3f28" (UID: "4402b7e2-fc19-4538-abdd-a39c587c3f28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:44.078829 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.078808 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4" pod "4402b7e2-fc19-4538-abdd-a39c587c3f28" (UID: "4402b7e2-fc19-4538-abdd-a39c587c3f28"). InnerVolumeSpecName "kube-api-access-7xqg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:44.078901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.078878 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca" pod "4402b7e2-fc19-4538-abdd-a39c587c3f28" (UID: "4402b7e2-fc19-4538-abdd-a39c587c3f28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:44.177498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.177422 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xqg4\" (UniqueName: \"kubernetes.io/projected/4402b7e2-fc19-4538-abdd-a39c587c3f28-kube-api-access-7xqg4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:44.177498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.177449 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:44.177498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.177459 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4402b7e2-fc19-4538-abdd-a39c587c3f28-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:44.844285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.844235 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" event={"ID":"4402b7e2-fc19-4538-abdd-a39c587c3f28","Type":"ContainerDied","Data":"fbe0daed1d022001f0835330702a4a60fd0551eb585b5be64977411623fe8374"} Nov 01 06:24:44.844285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.844285 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr" Nov 01 06:24:44.844504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:44.844304 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe0daed1d022001f0835330702a4a60fd0551eb585b5be64977411623fe8374" Nov 01 06:24:46.047929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:46.047895 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz"] Nov 01 06:24:46.098965 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:46.098933 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wp9xz"] Nov 01 06:24:46.971694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:46.971607 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22861021-78dc-44f5-aeed-fd4198b260e7" path="/var/lib/kubelet/pods/22861021-78dc-44f5-aeed-fd4198b260e7/volumes" Nov 01 06:24:56.453538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.453505 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:24:56.463987 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.463949 4425 cpu_manager.go:414] "containerMap: removing container" podUID="4402b7e2-fc19-4538-abdd-a39c587c3f28" containerName="revoke-keys" Nov 01 06:24:56.464224 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.464206 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4402b7e2-fc19-4538-abdd-a39c587c3f28" containerName="revoke-keys" Nov 01 06:24:56.464584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.464566 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4402b7e2-fc19-4538-abdd-a39c587c3f28" containerName="revoke-keys" Nov 01 06:24:56.468362 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.468339 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.521871 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.521823 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdwf\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.522177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522151 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.522461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522439 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.522641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522622 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.522784 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522765 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.522917 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522899 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.523053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.523037 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.523217 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.523199 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.523342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.522655 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-42db1c-openshift-default-dockercfg-8l2xz\"" Nov 01 06:24:56.523477 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.523444 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.615407 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.615366 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:24:56.626695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.626659 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.626988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.626966 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627104 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627227 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627389 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627562 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627545 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627668 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627655 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627774 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdwf\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.627898 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.627884 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.628843 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.628819 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.631925 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.629659 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.632104 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.629882 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.632230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.630094 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.632371 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.630552 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.635728 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.635680 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.640923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.640868 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.695427 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.695375 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.695589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.695549 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdwf\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\") pod \"e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.804306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:56.804189 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:56.947984 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:24:56.947940 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:24:57.241451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.241422 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:24:57.243128 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:24:57.243101 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2933d68_49b2_4bad_bf6f_15facc8fa05d.slice/crio-36aa43703bd9107101cd71d9c71c250de8faac13858f2bbc01f27237a78922a2 WatchSource:0}: Error finding container 36aa43703bd9107101cd71d9c71c250de8faac13858f2bbc01f27237a78922a2: Status 404 returned error can't find the container with id 36aa43703bd9107101cd71d9c71c250de8faac13858f2bbc01f27237a78922a2 Nov 01 06:24:57.245324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.245288 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:24:57.245421 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.245357 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:24:57.245421 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.245400 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:24:57.941951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.941914 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerStarted","Data":"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8"} Nov 01 06:24:57.942395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:57.941967 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerStarted","Data":"36aa43703bd9107101cd71d9c71c250de8faac13858f2bbc01f27237a78922a2"} Nov 01 06:24:58.048498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.048470 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt"] Nov 01 06:24:58.050437 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.050417 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bb5pt"] Nov 01 06:24:58.096369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.096319 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podStartSLOduration=2.096304365 podStartE2EDuration="2.096304365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:58.063152526 +0000 UTC m=+4913.203741446" watchObservedRunningTime="2026-11-01 06:24:58.096304365 +0000 UTC m=+4913.236893284" Nov 01 06:24:58.804809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.804769 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:58.810381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.810352 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:58.947173 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.947134 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:58.948134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.948116 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:24:58.972210 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:24:58.972186 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e32eea5-36b3-4a97-9dd9-cebd4941eb09" path="/var/lib/kubelet/pods/9e32eea5-36b3-4a97-9dd9-cebd4941eb09/volumes" Nov 01 06:25:04.652601 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.652561 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:25:04.656652 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.656627 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.713237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.713197 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gqt\" (UniqueName: \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.713442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.713274 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.714485 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.714461 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:25:04.761362 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.761329 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-42db1c\"" Nov 01 06:25:04.761546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.761426 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4fba256d-dockercfg-d58wn\"" Nov 01 06:25:04.814839 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.814805 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gqt\" (UniqueName: \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.815024 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.814847 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.815655 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.815632 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.876877 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.876841 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gqt\" (UniqueName: \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\") pod \"payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:04.969841 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:04.969750 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:05.344907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:05.344875 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:25:05.346071 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:25:05.346043 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00350ec5_9307_4b7d_84d8_4b706551b715.slice/crio-6362504bbd890eede5d3f5c0fd4c0b0e1023b517a6e3edd49a9612f1dbd7c9bb WatchSource:0}: Error finding container 6362504bbd890eede5d3f5c0fd4c0b0e1023b517a6e3edd49a9612f1dbd7c9bb: Status 404 returned error can't find the container with id 6362504bbd890eede5d3f5c0fd4c0b0e1023b517a6e3edd49a9612f1dbd7c9bb Nov 01 06:25:05.984387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:05.984348 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" event={"ID":"00350ec5-9307-4b7d-84d8-4b706551b715","Type":"ContainerStarted","Data":"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd"} Nov 01 06:25:05.984815 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:05.984512 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:05.984815 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:05.984551 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" event={"ID":"00350ec5-9307-4b7d-84d8-4b706551b715","Type":"ContainerStarted","Data":"6362504bbd890eede5d3f5c0fd4c0b0e1023b517a6e3edd49a9612f1dbd7c9bb"} Nov 01 06:25:06.102593 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:06.102535 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" podStartSLOduration=2.10251562 podStartE2EDuration="2.10251562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:06.102039404 +0000 UTC m=+4921.242628321" watchObservedRunningTime="2026-11-01 06:25:06.10251562 +0000 UTC m=+4921.243104540" Nov 01 06:25:16.992411 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:16.992379 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:25:24.103045 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.103010 4425 generic.go:356] "Generic (PLEG): container finished" podID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerID="527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8" exitCode=0 Nov 01 06:25:24.103559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.103085 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerDied","Data":"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8"} Nov 01 06:25:24.103559 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.103510 4425 scope.go:122] "RemoveContainer" containerID="527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8" Nov 01 06:25:24.104797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.104771 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:25:24.104882 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.104842 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:25:24.104882 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:24.104874 4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 01 06:25:25.115163 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:25.115127 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerStarted","Data":"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2"} Nov 01 06:25:25.804830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:25.804790 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:25:25.805844 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:25.805822 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" start-of-body= Nov 01 06:25:25.805910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:25.805883 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" Nov 01 06:25:26.804627 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:26.804586 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" start-of-body= Nov 01 06:25:26.805087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:26.804653 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" Nov 01 06:25:26.973298 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:25:26.973223 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:25:27.805432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:27.805394 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" start-of-body= Nov 01 06:25:27.805896 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:27.805453 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" Nov 01 06:25:28.771978 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.771943 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:25:28.774635 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.774612 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797452 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797410 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797628 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797474 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797628 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797567 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797731 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797648 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xxt\" (UniqueName: \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797782 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797765 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.797839 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.797820 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.805340 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.805312 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" start-of-body= Nov 01 06:25:28.805526 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.805370 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" Nov 01 06:25:28.828149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.828122 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-42db1c\"/\"default-dockercfg-wczq9\"" Nov 01 06:25:28.879282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.879143 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-42db1c\"/\"openshift-service-ca.crt\"" Nov 01 06:25:28.879282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.879179 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-42db1c\"/\"kube-root-ca.crt\"" Nov 01 06:25:28.882115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.882088 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:25:28.898527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898498 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.898664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898537 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.898664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898555 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.898664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898594 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xxt\" (UniqueName: \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.898664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898626 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.898664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898646 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.899004 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.898980 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.899080 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.899000 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.899136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.899081 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.900998 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.900979 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.928633 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.928592 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-42db1c\"/\"simulator-single-main-42db1c-kserve-self-signed-certs\"" Nov 01 06:25:28.932464 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.932428 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.970740 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.970706 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xxt\" (UniqueName: \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\") pod \"simulator-single-main-42db1c-kserve-8479f7784b-hthdp\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:28.973899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:28.973873 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:25:29.086774 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:29.086735 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:29.479535 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:29.479508 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:25:29.481171 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:25:29.481144 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9276f83_6e68_4644_80a4_3fcf4de20260.slice/crio-988ec924f5d624c443c674606608eaa4f04ef2ceed5788a0e9d264f6f707122c WatchSource:0}: Error finding container 988ec924f5d624c443c674606608eaa4f04ef2ceed5788a0e9d264f6f707122c: Status 404 returned error can't find the container with id 988ec924f5d624c443c674606608eaa4f04ef2ceed5788a0e9d264f6f707122c Nov 01 06:25:29.805640 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:29.805551 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" start-of-body= Nov 01 06:25:29.805640 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:29.805621 4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.90:15021/healthz/ready\": dial tcp 10.133.1.90:15021: connect: connection refused" Nov 01 06:25:30.142951 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:30.142915 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" event={"ID":"e9276f83-6e68-4644-80a4-3fcf4de20260","Type":"ContainerStarted","Data":"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead"} Nov 01 06:25:30.143220 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:30.142962 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" event={"ID":"e9276f83-6e68-4644-80a4-3fcf4de20260","Type":"ContainerStarted","Data":"988ec924f5d624c443c674606608eaa4f04ef2ceed5788a0e9d264f6f707122c"} Nov 01 06:25:30.262607 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:30.262553 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" podStartSLOduration=2.262538977 podStartE2EDuration="2.262538977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:30.261317666 +0000 UTC m=+4945.401906586" watchObservedRunningTime="2026-11-01 06:25:30.262538977 +0000 UTC m=+4945.403127935" Nov 01 06:25:30.809287 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:30.809236 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:25:30.810349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:30.810328 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:25:31.087685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:31.087653 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:33.046988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:33.046954 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd"] Nov 01 06:25:33.097708 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:33.097676 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tsdjd"] Nov 01 06:25:34.972077 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:34.972040 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ac426-dd35-4226-b239-ffe0a96dad65" path="/var/lib/kubelet/pods/f58ac426-dd35-4226-b239-ffe0a96dad65/volumes" Nov 01 06:25:39.087560 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:39.087523 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:39.100314 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:39.100287 4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:39.203188 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:39.203161 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:25:44.051869 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:44.051829 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm"] Nov 01 06:25:44.102186 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:44.102153 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rh7jm"] Nov 01 06:25:44.972334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:25:44.972299 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3245793a-2524-496a-bb64-1bca4f2d373a" path="/var/lib/kubelet/pods/3245793a-2524-496a-bb64-1bca4f2d373a/volumes" Nov 01 06:25:56.948034 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:25:56.947992 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:26:03.599341 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.599305 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7"] Nov 01 06:26:03.603188 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.603172 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.657122 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.657082 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.657492 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.657174 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkgg\" (UniqueName: \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.657492 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.657216 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.657671 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.657581 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:26:03.708592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.708554 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:26:03.708592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.708602 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:26:03.708855 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.708635 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:03.711060 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.711035 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7"] Nov 01 06:26:03.758376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.758341 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.758552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.758474 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.758552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.758537 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkgg\" (UniqueName: \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.759010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.758989 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.761230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.761178 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.820779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.820740 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkgg\" (UniqueName: \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\") pod \"maas-api-revoke-sub-456adb91-tk7k7\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:03.920448 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:03.920360 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:04.294926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:04.294899 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7"] Nov 01 06:26:04.296381 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:26:04.296351 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbecd75b_6fcd_4713_8f51_21272da8b7bf.slice/crio-b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4 WatchSource:0}: Error finding container b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4: Status 404 returned error can't find the container with id b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4 Nov 01 06:26:04.331921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:04.331891 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" event={"ID":"cbecd75b-6fcd-4713-8f51-21272da8b7bf","Type":"ContainerStarted","Data":"b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4"} Nov 01 06:26:05.339084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:05.339050 4425 generic.go:356] "Generic (PLEG): container finished" podID="cbecd75b-6fcd-4713-8f51-21272da8b7bf" containerID="0b1daa6d3aab06e530bcfef40b1d08441df3fb97581c8d2a5cbc5293a84c9fdb" exitCode=0 Nov 01 06:26:05.339507 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:05.339099 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" event={"ID":"cbecd75b-6fcd-4713-8f51-21272da8b7bf","Type":"ContainerDied","Data":"0b1daa6d3aab06e530bcfef40b1d08441df3fb97581c8d2a5cbc5293a84c9fdb"} Nov 01 06:26:06.487089 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.487057 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:06.586872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.586837 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\") pod \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " Nov 01 06:26:06.587062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.586884 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\") pod \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " Nov 01 06:26:06.587062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.586979 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\" (UniqueName: \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\") pod \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\" (UID: \"cbecd75b-6fcd-4713-8f51-21272da8b7bf\") " Nov 01 06:26:06.587303 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.587245 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca" pod "cbecd75b-6fcd-4713-8f51-21272da8b7bf" (UID: "cbecd75b-6fcd-4713-8f51-21272da8b7bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:06.587376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.587362 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:06.592154 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.589883 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca" pod "cbecd75b-6fcd-4713-8f51-21272da8b7bf" (UID: "cbecd75b-6fcd-4713-8f51-21272da8b7bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:06.592154 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.591169 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg" pod "cbecd75b-6fcd-4713-8f51-21272da8b7bf" (UID: "cbecd75b-6fcd-4713-8f51-21272da8b7bf"). InnerVolumeSpecName "kube-api-access-tgkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:06.688219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.688184 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbecd75b-6fcd-4713-8f51-21272da8b7bf-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:06.688219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:06.688212 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkgg\" (UniqueName: \"kubernetes.io/projected/cbecd75b-6fcd-4713-8f51-21272da8b7bf-kube-api-access-tgkgg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:07.351750 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:07.351719 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" Nov 01 06:26:07.351950 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:07.351746 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7" event={"ID":"cbecd75b-6fcd-4713-8f51-21272da8b7bf","Type":"ContainerDied","Data":"b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4"} Nov 01 06:26:07.351950 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:07.351799 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b12c229728c0dcf288e189daa2b372ef4963493356cd5841c1de3e1e42f6c4" Nov 01 06:26:08.847338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:08.847311 4425 scope.go:122] "RemoveContainer" containerID="b527aff8aac166d0b60468ce1d9d2b411e83c84bd722d0207d06947c0e89fd28" Nov 01 06:26:08.859969 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:08.859947 4425 scope.go:122] "RemoveContainer" containerID="b6718153c1a3213f5743217a4b1cdd1f410dcf1785acc0ec991321dd74e2ef90" Nov 01 06:26:18.046310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:18.046194 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng"] Nov 01 06:26:18.096826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:18.096794 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-ttgng"] Nov 01 06:26:18.973574 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:18.973538 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cca638-4642-4e05-9b2d-5fb02a1fa894" path="/var/lib/kubelet/pods/23cca638-4642-4e05-9b2d-5fb02a1fa894/volumes" Nov 01 06:26:20.222088 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:20.222048 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:26:22.279071 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:22.279035 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:26:26.948760 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:26:26.948718 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:26:27.280114 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:27.280021 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" containerID="cri-o://c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd" gracePeriod=25 Nov 01 06:26:35.245835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.245774 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" podUID="e9276f83-6e68-4644-80a4-3fcf4de20260" containerName="main" containerID="cri-o://ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead" gracePeriod=45 Nov 01 06:26:35.506346 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.506275 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:26:35.508827 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.508798 4425 generic.go:356] "Generic (PLEG): container finished" podID="e9276f83-6e68-4644-80a4-3fcf4de20260" containerID="ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead" exitCode=0 Nov 01 06:26:35.508959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.508835 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" Nov 01 06:26:35.508959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.508892 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" event={"ID":"e9276f83-6e68-4644-80a4-3fcf4de20260","Type":"ContainerDied","Data":"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead"} Nov 01 06:26:35.508959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.508931 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp" event={"ID":"e9276f83-6e68-4644-80a4-3fcf4de20260","Type":"ContainerDied","Data":"988ec924f5d624c443c674606608eaa4f04ef2ceed5788a0e9d264f6f707122c"} Nov 01 06:26:35.508959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.508953 4425 scope.go:122] "RemoveContainer" containerID="ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead" Nov 01 06:26:35.518115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.518099 4425 scope.go:122] "RemoveContainer" containerID="ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead" Nov 01 06:26:35.518431 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:26:35.518412 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead\": container with ID starting with ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead not found: ID does not exist" containerID="ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead" Nov 01 06:26:35.518506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.518437 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead"} err="failed to get container status \"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead\": rpc error: code = NotFound desc = could not find container \"ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead\": container with ID starting with ba3d2c8b3daf0adf71e262dba6f4a526d38124c88c27bb29c7be85f061c81ead not found: ID does not exist" Nov 01 06:26:35.583419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583388 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583422 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583687 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583494 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583687 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583541 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583687 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583571 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583687 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583606 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\" (UniqueName: \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\") pod \"e9276f83-6e68-4644-80a4-3fcf4de20260\" (UID: \"e9276f83-6e68-4644-80a4-3fcf4de20260\") " Nov 01 06:26:35.583924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583773 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:35.583924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583788 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:35.583924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.583899 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:35.584088 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.584067 4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-tmp-dir\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.584143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.584088 4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-home\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.584143 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.584102 4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-model-cache\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.585974 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.585948 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:35.585974 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.585964 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:35.586135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.585954 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt" pod "e9276f83-6e68-4644-80a4-3fcf4de20260" (UID: "e9276f83-6e68-4644-80a4-3fcf4de20260"). InnerVolumeSpecName "kube-api-access-68xxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:35.685510 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.685482 4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9276f83-6e68-4644-80a4-3fcf4de20260-dshm\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.685713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.685520 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68xxt\" (UniqueName: \"kubernetes.io/projected/e9276f83-6e68-4644-80a4-3fcf4de20260-kube-api-access-68xxt\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.685713 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.685532 4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9276f83-6e68-4644-80a4-3fcf4de20260-tls-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:35.933022 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.932980 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:26:35.936710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:35.936682 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-42db1c/simulator-single-main-42db1c-kserve-8479f7784b-hthdp"] Nov 01 06:26:36.972234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:36.972201 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9276f83-6e68-4644-80a4-3fcf4de20260" path="/var/lib/kubelet/pods/e9276f83-6e68-4644-80a4-3fcf4de20260/volumes" Nov 01 06:26:36.990297 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:36.990240 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:26:40.047391 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:40.047355 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4"] Nov 01 06:26:40.098110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:40.098070 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-54sl4"] Nov 01 06:26:40.972371 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:40.972327 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb7c4f9-857f-487e-8c90-a9268e3f1350" path="/var/lib/kubelet/pods/bdb7c4f9-857f-487e-8c90-a9268e3f1350/volumes" Nov 01 06:26:46.990889 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:46.990848 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:26:47.543699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.543592 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:26:47.584059 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.584018 4425 generic.go:356] "Generic (PLEG): container finished" podID="00350ec5-9307-4b7d-84d8-4b706551b715" containerID="c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd" exitCode=0 Nov 01 06:26:47.584310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.584090 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" Nov 01 06:26:47.584310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.584132 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" event={"ID":"00350ec5-9307-4b7d-84d8-4b706551b715","Type":"ContainerDied","Data":"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd"} Nov 01 06:26:47.584310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.584208 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl" event={"ID":"00350ec5-9307-4b7d-84d8-4b706551b715","Type":"ContainerDied","Data":"6362504bbd890eede5d3f5c0fd4c0b0e1023b517a6e3edd49a9612f1dbd7c9bb"} Nov 01 06:26:47.584310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.584222 4425 scope.go:122] "RemoveContainer" containerID="c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd" Nov 01 06:26:47.593722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.593689 4425 scope.go:122] "RemoveContainer" containerID="c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd" Nov 01 06:26:47.594207 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:26:47.594169 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd\": container with ID starting with c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd not found: ID does not exist" containerID="c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd" Nov 01 06:26:47.594302 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.594234 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd"} err="failed to get container status \"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd\": rpc error: code = NotFound desc = could not find container \"c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd\": container with ID starting with c3cdc6f87e876c8824a7d9c37b3c9fae1820a810d0488cb8815456015b022ebd not found: ID does not exist" Nov 01 06:26:47.601744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.601710 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\") pod \"00350ec5-9307-4b7d-84d8-4b706551b715\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " Nov 01 06:26:47.601927 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.601841 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\" (UniqueName: \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\") pod \"00350ec5-9307-4b7d-84d8-4b706551b715\" (UID: \"00350ec5-9307-4b7d-84d8-4b706551b715\") " Nov 01 06:26:47.602118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.602092 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume" pod "00350ec5-9307-4b7d-84d8-4b706551b715" (UID: "00350ec5-9307-4b7d-84d8-4b706551b715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:47.604208 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.604173 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt" pod "00350ec5-9307-4b7d-84d8-4b706551b715" (UID: "00350ec5-9307-4b7d-84d8-4b706551b715"). InnerVolumeSpecName "kube-api-access-f5gqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:47.703744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.703699 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gqt\" (UniqueName: \"kubernetes.io/projected/00350ec5-9307-4b7d-84d8-4b706551b715-kube-api-access-f5gqt\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:47.703744 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:47.703731 4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00350ec5-9307-4b7d-84d8-4b706551b715-plugins-config-volume\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:48.005914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:48.005878 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:26:48.009698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:48.009671 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42db1c-569dbrhmgl"] Nov 01 06:26:48.973181 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:48.973150 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" path="/var/lib/kubelet/pods/00350ec5-9307-4b7d-84d8-4b706551b715/volumes" Nov 01 06:26:56.947154 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:26:56.947114 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:26:59.857735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:59.857700 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:26:59.858284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:26:59.857966 4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" containerID="cri-o://e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2" gracePeriod=30 Nov 01 06:27:00.812098 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:00.812058 4425 patch_prober.go:29] interesting pod/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:27:00.812304 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:00.812117 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:27:01.411197 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.411161 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr"] Nov 01 06:27:01.461434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.461398 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wdlcr"] Nov 01 06:27:01.534529 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.534492 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t"] Nov 01 06:27:01.534971 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.534955 4425 cpu_manager.go:414] "containerMap: removing container" podUID="e9276f83-6e68-4644-80a4-3fcf4de20260" containerName="main" Nov 01 06:27:01.535017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.534974 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9276f83-6e68-4644-80a4-3fcf4de20260" containerName="main" Nov 01 06:27:01.535017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.534996 4425 cpu_manager.go:414] "containerMap: removing container" podUID="cbecd75b-6fcd-4713-8f51-21272da8b7bf" containerName="revoke-keys" Nov 01 06:27:01.535017 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535002 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecd75b-6fcd-4713-8f51-21272da8b7bf" containerName="revoke-keys" Nov 01 06:27:01.535120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535046 4425 cpu_manager.go:414] "containerMap: removing container" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" Nov 01 06:27:01.535120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535052 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" Nov 01 06:27:01.535183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535124 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="00350ec5-9307-4b7d-84d8-4b706551b715" containerName="payload-pre-processing" Nov 01 06:27:01.535183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535134 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9276f83-6e68-4644-80a4-3fcf4de20260" containerName="main" Nov 01 06:27:01.535183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.535140 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecd75b-6fcd-4713-8f51-21272da8b7bf" containerName="revoke-keys" Nov 01 06:27:01.538486 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.538469 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.593349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.593310 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:27:01.593552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.593420 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:27:01.593552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.593467 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:27:01.593712 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.593324 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:27:01.596634 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.596609 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t"] Nov 01 06:27:01.646348 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.646315 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.646532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.646360 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.646532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.646449 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zzk\" (UniqueName: \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.747923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.747830 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.747923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.747877 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.747923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.747912 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zzk\" (UniqueName: \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.748575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.748547 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.750527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.750507 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.810191 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.810157 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zzk\" (UniqueName: \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\") pod \"maas-api-revoke-sub-3d4156f1-4hc4t\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:01.851954 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:01.851920 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:02.225953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:02.225927 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t"] Nov 01 06:27:02.227826 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:27:02.227798 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71cd289_a048_4f7d_a086_cf611204dbe2.slice/crio-22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832 WatchSource:0}: Error finding container 22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832: Status 404 returned error can't find the container with id 22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832 Nov 01 06:27:02.665365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:02.665314 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" event={"ID":"b71cd289-a048-4f7d-a086-cf611204dbe2","Type":"ContainerStarted","Data":"817a3ea6fe750b184e731b6644d24eac83bd4b69d87ef5788f5bb1d565413fe9"} Nov 01 06:27:02.665837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:02.665394 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" event={"ID":"b71cd289-a048-4f7d-a086-cf611204dbe2","Type":"ContainerStarted","Data":"22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832"} Nov 01 06:27:02.784194 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:02.784139 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" podStartSLOduration=1.784126602 podStartE2EDuration="1.784126602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:02.782402953 +0000 UTC m=+5037.922991877" watchObservedRunningTime="2026-11-01 06:27:02.784126602 +0000 UTC m=+5037.924715521" Nov 01 06:27:02.972689 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:02.972606 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4402b7e2-fc19-4538-abdd-a39c587c3f28" path="/var/lib/kubelet/pods/4402b7e2-fc19-4538-abdd-a39c587c3f28/volumes" Nov 01 06:27:03.673733 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:03.673699 4425 generic.go:356] "Generic (PLEG): container finished" podID="b71cd289-a048-4f7d-a086-cf611204dbe2" containerID="817a3ea6fe750b184e731b6644d24eac83bd4b69d87ef5788f5bb1d565413fe9" exitCode=0 Nov 01 06:27:03.674175 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:03.673748 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" event={"ID":"b71cd289-a048-4f7d-a086-cf611204dbe2","Type":"ContainerDied","Data":"817a3ea6fe750b184e731b6644d24eac83bd4b69d87ef5788f5bb1d565413fe9"} Nov 01 06:27:04.822314 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.822284 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:04.982610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.982582 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\") pod \"b71cd289-a048-4f7d-a086-cf611204dbe2\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " Nov 01 06:27:04.982759 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.982631 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\") pod \"b71cd289-a048-4f7d-a086-cf611204dbe2\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " Nov 01 06:27:04.982759 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.982719 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\" (UniqueName: \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\") pod \"b71cd289-a048-4f7d-a086-cf611204dbe2\" (UID: \"b71cd289-a048-4f7d-a086-cf611204dbe2\") " Nov 01 06:27:04.983040 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.983017 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca" pod "b71cd289-a048-4f7d-a086-cf611204dbe2" (UID: "b71cd289-a048-4f7d-a086-cf611204dbe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:04.984990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.984965 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca" pod "b71cd289-a048-4f7d-a086-cf611204dbe2" (UID: "b71cd289-a048-4f7d-a086-cf611204dbe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:27:04.985180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:04.985160 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk" pod "b71cd289-a048-4f7d-a086-cf611204dbe2" (UID: "b71cd289-a048-4f7d-a086-cf611204dbe2"). InnerVolumeSpecName "kube-api-access-d6zzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:05.083858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.083821 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.083858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.083855 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71cd289-a048-4f7d-a086-cf611204dbe2-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.083858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.083866 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zzk\" (UniqueName: \"kubernetes.io/projected/b71cd289-a048-4f7d-a086-cf611204dbe2-kube-api-access-d6zzk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.103176 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.103149 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:27:05.184489 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184456 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184522 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184546 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184620 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184690 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184652 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184709 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184814 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184862 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.184912 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184897 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\") pod \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\" (UID: \"a2933d68-49b2-4bad-bf6f-15facc8fa05d\") " Nov 01 06:27:05.185140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.184938 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.185199 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185185 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.185387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185362 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.185387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185375 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.185593 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185556 4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.185686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185600 4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-data\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.185751 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.185728 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:05.187218 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.187194 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:27:05.187395 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.187229 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "kube-api-access-7xdwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:05.187662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.187636 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:05.187662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.187649 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token" pod "a2933d68-49b2-4bad-bf6f-15facc8fa05d" (UID: "a2933d68-49b2-4bad-bf6f-15facc8fa05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286724 4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-workload-certs\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286754 4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istiod-ca-cert\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286764 4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-podinfo\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286772 4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-envoy\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286782 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xdwf\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-kube-api-access-7xdwf\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286792 4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2933d68-49b2-4bad-bf6f-15facc8fa05d-istio-token\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.286810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.286801 4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2933d68-49b2-4bad-bf6f-15facc8fa05d-credential-socket\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:05.686142 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.686108 4425 generic.go:356] "Generic (PLEG): container finished" podID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerID="e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2" exitCode=0 Nov 01 06:27:05.686373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.686160 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" Nov 01 06:27:05.686373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.686186 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerDied","Data":"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2"} Nov 01 06:27:05.686373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.686274 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw" event={"ID":"a2933d68-49b2-4bad-bf6f-15facc8fa05d","Type":"ContainerDied","Data":"36aa43703bd9107101cd71d9c71c250de8faac13858f2bbc01f27237a78922a2"} Nov 01 06:27:05.686373 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.686292 4425 scope.go:122] "RemoveContainer" containerID="e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2" Nov 01 06:27:05.687754 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.687730 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" Nov 01 06:27:05.687953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.687808 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t" event={"ID":"b71cd289-a048-4f7d-a086-cf611204dbe2","Type":"ContainerDied","Data":"22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832"} Nov 01 06:27:05.687994 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.687983 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22194a116ed0e7f72361252a0174dfc6918bf03d5eab72af578403de66f73832" Nov 01 06:27:05.696878 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.696857 4425 scope.go:122] "RemoveContainer" containerID="527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8" Nov 01 06:27:05.705415 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.705398 4425 scope.go:122] "RemoveContainer" containerID="e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2" Nov 01 06:27:05.705684 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:27:05.705665 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2\": container with ID starting with e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2 not found: ID does not exist" containerID="e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2" Nov 01 06:27:05.705766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.705693 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2"} err="failed to get container status \"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2\": rpc error: code = NotFound desc = could not find container \"e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2\": container with ID starting with e86ab02822eab4a986638c8b9922f5a794b955c06d17438052f3ac4cf26104c2 not found: ID does not exist" Nov 01 06:27:05.705766 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.705709 4425 scope.go:122] "RemoveContainer" containerID="527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8" Nov 01 06:27:05.705979 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:27:05.705957 4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8\": container with ID starting with 527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8 not found: ID does not exist" containerID="527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8" Nov 01 06:27:05.706087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.705989 4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8"} err="failed to get container status \"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8\": rpc error: code = NotFound desc = could not find container \"527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8\": container with ID starting with 527286df430833dda8971c6762f7e4e41f4aeb127e1a4063ef63c5494ac347c8 not found: ID does not exist" Nov 01 06:27:05.854157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.854127 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:27:05.874811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:05.874778 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42db1c-openshift-default-6bd6c77dd98qhcw"] Nov 01 06:27:06.972108 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:06.972077 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" path="/var/lib/kubelet/pods/a2933d68-49b2-4bad-bf6f-15facc8fa05d/volumes" Nov 01 06:27:08.885571 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:08.885540 4425 scope.go:122] "RemoveContainer" containerID="202312de94765f02f163ff03b1f57cdbf6afac3bd768803d3dbdc4518660425c" Nov 01 06:27:08.894649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:08.894622 4425 scope.go:122] "RemoveContainer" containerID="77100cd27a55e0a1366703ebd58fb65cfdfbbf8fce85e0172dda593c6f62e03c" Nov 01 06:27:09.046746 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:09.046714 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh"] Nov 01 06:27:09.097735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:09.097704 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-p4qbh"] Nov 01 06:27:10.973291 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:10.973237 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5df34c7-8367-400c-b190-f3f7411b0fc2" path="/var/lib/kubelet/pods/c5df34c7-8367-400c-b190-f3f7411b0fc2/volumes" Nov 01 06:27:17.824461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:17.824416 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-8ng9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:27:17.825318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:17.825280 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-8ng9q" podUID="642bdc4b-db0b-4f67-aab4-aa429c34b921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:27:17.957280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:17.953357 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:27:17.957280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:17.953452 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:27:26.047356 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:26.047319 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2"] Nov 01 06:27:26.097815 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:26.097785 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7d9v2"] Nov 01 06:27:26.947368 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:27:26.947334 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:27:26.973068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:27:26.973032 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc6c8cd-6601-4985-a9d1-69f1e83cb971" path="/var/lib/kubelet/pods/2cc6c8cd-6601-4985-a9d1-69f1e83cb971/volumes" Nov 01 06:27:56.946999 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:27:56.946917 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:28:08.931557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:08.931525 4425 scope.go:122] "RemoveContainer" containerID="6e0f5d1e01412a7ee057b017d88f63a55f904e3383ac436267af1764c6c6983a" Nov 01 06:28:08.942854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:08.942827 4425 scope.go:122] "RemoveContainer" containerID="0a08d4ef7d436f5ac8c16f9f832642b94d05434f8fe8a424286aec0c9becef27" Nov 01 06:28:08.951646 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:08.951626 4425 scope.go:122] "RemoveContainer" containerID="ad3febeaa6df9b50da6e615f6736d2fb299d4d413277c39041f3943aab5c06e9" Nov 01 06:28:26.946732 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:28:26.946687 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:28:46.053278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:46.050928 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65"] Nov 01 06:28:46.108278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:46.104940 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5w65"] Nov 01 06:28:46.933423 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:46.933377 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-b4bkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:28:46.933617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:46.933444 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-b4bkr" podUID="61a296e9-2e08-439f-96cc-e2c46a96a22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:46.991444 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:46.991407 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486154bf-ceea-4f30-9803-8f2f0a265b74" path="/var/lib/kubelet/pods/486154bf-ceea-4f30-9803-8f2f0a265b74/volumes" Nov 01 06:28:51.306847 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.306804 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf"] Nov 01 06:28:51.307428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307409 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b71cd289-a048-4f7d-a086-cf611204dbe2" containerName="revoke-keys" Nov 01 06:28:51.307527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307431 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71cd289-a048-4f7d-a086-cf611204dbe2" containerName="revoke-keys" Nov 01 06:28:51.307527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307452 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:28:51.307527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307459 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:28:51.307527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307490 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:28:51.307527 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307498 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:28:51.307789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307602 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:28:51.307789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.307622 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71cd289-a048-4f7d-a086-cf611204dbe2" containerName="revoke-keys" Nov 01 06:28:51.310805 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.310784 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.365457 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.365420 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:28:51.378044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.378008 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.378232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.378073 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.378232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.378195 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.416272 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.416218 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:28:51.416272 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.416216 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:28:51.416515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.416471 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:28:51.418438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.418413 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf"] Nov 01 06:28:51.479583 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.479548 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.479764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.479589 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.479764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.479723 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.480328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.480306 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.482043 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.482017 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.542555 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.542514 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\") pod \"maas-api-revoke-sub-d576e68b-hkwgf\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.623165 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.623138 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:51.998309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:51.998286 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf"] Nov 01 06:28:52.000461 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:28:52.000434 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99016e8c_0952_49cd_90a8_fbd7be744f5b.slice/crio-742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1 WatchSource:0}: Error finding container 742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1: Status 404 returned error can't find the container with id 742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1 Nov 01 06:28:52.314407 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:52.314365 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" event={"ID":"99016e8c-0952-49cd-90a8-fbd7be744f5b","Type":"ContainerStarted","Data":"00ac0faef68f45b22d15da4b530e6531d3c1b3b91e763c063bdd175e36c099b5"} Nov 01 06:28:52.314812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:52.314433 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" event={"ID":"99016e8c-0952-49cd-90a8-fbd7be744f5b","Type":"ContainerStarted","Data":"742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1"} Nov 01 06:28:52.432342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:52.432267 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" podStartSLOduration=1.432234358 podStartE2EDuration="1.432234358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:52.43137953 +0000 UTC m=+5147.571968448" watchObservedRunningTime="2026-11-01 06:28:52.432234358 +0000 UTC m=+5147.572823276" Nov 01 06:28:53.320560 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:53.320529 4425 generic.go:356] "Generic (PLEG): container finished" podID="99016e8c-0952-49cd-90a8-fbd7be744f5b" containerID="00ac0faef68f45b22d15da4b530e6531d3c1b3b91e763c063bdd175e36c099b5" exitCode=0 Nov 01 06:28:53.320982 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:53.320613 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" event={"ID":"99016e8c-0952-49cd-90a8-fbd7be744f5b","Type":"ContainerDied","Data":"00ac0faef68f45b22d15da4b530e6531d3c1b3b91e763c063bdd175e36c099b5"} Nov 01 06:28:54.463347 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.463317 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:54.509759 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.509730 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\") pod \"99016e8c-0952-49cd-90a8-fbd7be744f5b\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " Nov 01 06:28:54.509918 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.509824 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\") pod \"99016e8c-0952-49cd-90a8-fbd7be744f5b\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " Nov 01 06:28:54.509918 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.509900 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\") pod \"99016e8c-0952-49cd-90a8-fbd7be744f5b\" (UID: \"99016e8c-0952-49cd-90a8-fbd7be744f5b\") " Nov 01 06:28:54.510246 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.510204 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca" pod "99016e8c-0952-49cd-90a8-fbd7be744f5b" (UID: "99016e8c-0952-49cd-90a8-fbd7be744f5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:54.512171 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.512149 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca" pod "99016e8c-0952-49cd-90a8-fbd7be744f5b" (UID: "99016e8c-0952-49cd-90a8-fbd7be744f5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:54.512171 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.512154 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5" pod "99016e8c-0952-49cd-90a8-fbd7be744f5b" (UID: "99016e8c-0952-49cd-90a8-fbd7be744f5b"). InnerVolumeSpecName "kube-api-access-5bpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:54.611617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.611535 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:54.611617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.611562 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpm5\" (UniqueName: \"kubernetes.io/projected/99016e8c-0952-49cd-90a8-fbd7be744f5b-kube-api-access-5bpm5\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:54.611617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:54.611573 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99016e8c-0952-49cd-90a8-fbd7be744f5b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:55.334329 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:55.334292 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" event={"ID":"99016e8c-0952-49cd-90a8-fbd7be744f5b","Type":"ContainerDied","Data":"742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1"} Nov 01 06:28:55.334518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:55.334343 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="742f397d58a5fa2f3f41fe7d768e4ead05a61fb6e62a09a23d3acd2d1cf739f1" Nov 01 06:28:55.334518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:55.334431 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf" Nov 01 06:28:56.947819 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:28:56.947778 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:28:57.045760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:57.045726 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5"] Nov 01 06:28:57.096552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:57.096521 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4r4d5"] Nov 01 06:28:58.972538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:28:58.972496 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26da6e7a-0b3a-45cd-9869-90cbd83fcd0a" path="/var/lib/kubelet/pods/26da6e7a-0b3a-45cd-9869-90cbd83fcd0a/volumes" Nov 01 06:29:08.978555 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:08.978524 4425 scope.go:122] "RemoveContainer" containerID="f5a178e5448079f46224b25a5d0092bda914d8bbc11bb029018f85fcaddec07e" Nov 01 06:29:08.987223 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:08.987204 4425 scope.go:122] "RemoveContainer" containerID="79c4e09fd8dc407a463e0030229b5a81f460db6667cbc2cf886082dbfa74c865" Nov 01 06:29:08.995150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:08.995129 4425 scope.go:122] "RemoveContainer" containerID="0334d28f8e767d30978a36a485bd66ed1c6deb97fd0d9d9b83f990ded603e4b5" Nov 01 06:29:12.438014 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.437981 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq"] Nov 01 06:29:12.438522 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.438481 4425 cpu_manager.go:414] "containerMap: removing container" podUID="99016e8c-0952-49cd-90a8-fbd7be744f5b" containerName="revoke-keys" Nov 01 06:29:12.438522 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.438495 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99016e8c-0952-49cd-90a8-fbd7be744f5b" containerName="revoke-keys" Nov 01 06:29:12.438641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.438569 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99016e8c-0952-49cd-90a8-fbd7be744f5b" containerName="revoke-keys" Nov 01 06:29:12.438641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.438583 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2933d68-49b2-4bad-bf6f-15facc8fa05d" containerName="istio-proxy" Nov 01 06:29:12.443381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.443361 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.521580 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.521341 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq"] Nov 01 06:29:12.532698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.532661 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbw2f\" (UniqueName: \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.532698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.532712 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.532968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.532824 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.548174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.548139 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:29:12.548174 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.548139 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:29:12.548432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.548245 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:29:12.548432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.548273 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:29:12.633955 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.633920 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbw2f\" (UniqueName: \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.633955 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.633964 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.634235 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.634004 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.634787 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.634759 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.636709 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.636677 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.696440 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.696363 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbw2f\" (UniqueName: \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\") pod \"maas-api-revoke-sub-d2bc93bc-wb7lq\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:12.757244 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:12.757209 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:13.131886 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:13.131857 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq"] Nov 01 06:29:13.133995 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:29:13.133964 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff96cdb1_8abb_4cf1_9c90_03fbfc4f2f9d.slice/crio-8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0 WatchSource:0}: Error finding container 8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0: Status 404 returned error can't find the container with id 8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0 Nov 01 06:29:13.449663 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:13.449630 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" event={"ID":"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d","Type":"ContainerStarted","Data":"8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0"} Nov 01 06:29:14.456421 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:14.456386 4425 generic.go:356] "Generic (PLEG): container finished" podID="ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" containerID="5e835ebeff36f4dc9443eb9ae207f9c1adea51ac2b7d745b58f2b8d45b94268e" exitCode=0 Nov 01 06:29:14.456828 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:14.456474 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" event={"ID":"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d","Type":"ContainerDied","Data":"5e835ebeff36f4dc9443eb9ae207f9c1adea51ac2b7d745b58f2b8d45b94268e"} Nov 01 06:29:15.598809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.598782 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:15.665490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.665458 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\") pod \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " Nov 01 06:29:15.665683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.665563 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\") pod \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " Nov 01 06:29:15.665683 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.665607 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\" (UniqueName: \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\") pod \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\" (UID: \"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d\") " Nov 01 06:29:15.665929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.665905 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca" pod "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" (UID: "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:15.665996 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.665980 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:15.667557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.667537 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca" pod "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" (UID: "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:29:15.667709 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.667689 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f" pod "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" (UID: "ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d"). InnerVolumeSpecName "kube-api-access-gbw2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:15.767280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.767175 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbw2f\" (UniqueName: \"kubernetes.io/projected/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-kube-api-access-gbw2f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:15.767280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:15.767207 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:16.467106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:16.467031 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" Nov 01 06:29:16.467106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:16.467073 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq" event={"ID":"ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d","Type":"ContainerDied","Data":"8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0"} Nov 01 06:29:16.467379 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:16.467123 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8744987525f4f678632b5d5f2058ca0a44a960e1f036453f2130af708037d7a0" Nov 01 06:29:26.947080 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:29:26.947035 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:29:49.455492 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.455450 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z"] Nov 01 06:29:49.456149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.456091 4425 cpu_manager.go:414] "containerMap: removing container" podUID="ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" containerName="revoke-keys" Nov 01 06:29:49.456149 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.456112 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" containerName="revoke-keys" Nov 01 06:29:49.456318 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.456281 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" containerName="revoke-keys" Nov 01 06:29:49.459885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.459865 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.514230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.514179 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:29:49.553295 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.553242 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.553467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.553390 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvl9f\" (UniqueName: \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.553467 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.553419 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.564780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.564732 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:29:49.564780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.564748 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:29:49.564780 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.564782 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:29:49.568804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.568769 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z"] Nov 01 06:29:49.654020 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.653980 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvl9f\" (UniqueName: \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.654239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.654030 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.654239 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.654171 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.655119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.655042 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.656838 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.656812 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.716075 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.715994 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvl9f\" (UniqueName: \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\") pod \"maas-api-revoke-sub-a601598a-xqx7z\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:49.772859 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:49.772827 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:50.146124 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:50.146099 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z"] Nov 01 06:29:50.148537 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:29:50.148510 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd59dedc_a03e_408d_8d50_18402ebf9294.slice/crio-6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd WatchSource:0}: Error finding container 6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd: Status 404 returned error can't find the container with id 6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd Nov 01 06:29:50.666775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:50.666739 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" event={"ID":"dd59dedc-a03e-408d-8d50-18402ebf9294","Type":"ContainerStarted","Data":"7410e1d554413ce00978bf31617f71ef546f4574984df90e51719e9d8cd9e3a8"} Nov 01 06:29:50.667147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:50.666802 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" event={"ID":"dd59dedc-a03e-408d-8d50-18402ebf9294","Type":"ContainerStarted","Data":"6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd"} Nov 01 06:29:50.785367 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:50.785240 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" podStartSLOduration=1.785225125 podStartE2EDuration="1.785225125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:50.783788395 +0000 UTC m=+5205.924377313" watchObservedRunningTime="2026-11-01 06:29:50.785225125 +0000 UTC m=+5205.925814044" Nov 01 06:29:51.673133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:51.673097 4425 generic.go:356] "Generic (PLEG): container finished" podID="dd59dedc-a03e-408d-8d50-18402ebf9294" containerID="7410e1d554413ce00978bf31617f71ef546f4574984df90e51719e9d8cd9e3a8" exitCode=0 Nov 01 06:29:51.673579 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:51.673184 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" event={"ID":"dd59dedc-a03e-408d-8d50-18402ebf9294","Type":"ContainerDied","Data":"7410e1d554413ce00978bf31617f71ef546f4574984df90e51719e9d8cd9e3a8"} Nov 01 06:29:52.824798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.824762 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:52.886491 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.886459 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\" (UniqueName: \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\") pod \"dd59dedc-a03e-408d-8d50-18402ebf9294\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " Nov 01 06:29:52.886693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.886517 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\") pod \"dd59dedc-a03e-408d-8d50-18402ebf9294\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " Nov 01 06:29:52.886693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.886591 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\") pod \"dd59dedc-a03e-408d-8d50-18402ebf9294\" (UID: \"dd59dedc-a03e-408d-8d50-18402ebf9294\") " Nov 01 06:29:52.886962 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.886939 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca" pod "dd59dedc-a03e-408d-8d50-18402ebf9294" (UID: "dd59dedc-a03e-408d-8d50-18402ebf9294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:52.888806 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.888771 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca" pod "dd59dedc-a03e-408d-8d50-18402ebf9294" (UID: "dd59dedc-a03e-408d-8d50-18402ebf9294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:29:52.888922 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.888865 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f" pod "dd59dedc-a03e-408d-8d50-18402ebf9294" (UID: "dd59dedc-a03e-408d-8d50-18402ebf9294"). InnerVolumeSpecName "kube-api-access-dvl9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:52.987662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.987636 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvl9f\" (UniqueName: \"kubernetes.io/projected/dd59dedc-a03e-408d-8d50-18402ebf9294-kube-api-access-dvl9f\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:52.987662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.987663 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:52.987830 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:52.987672 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd59dedc-a03e-408d-8d50-18402ebf9294-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:53.684380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:53.684348 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" event={"ID":"dd59dedc-a03e-408d-8d50-18402ebf9294","Type":"ContainerDied","Data":"6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd"} Nov 01 06:29:53.684567 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:53.684397 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fe2349d3d337da96e0cd0299970ce3add9a7f3a75c5219dff3243b13f34a5bd" Nov 01 06:29:53.684567 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:29:53.684359 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z" Nov 01 06:29:56.948154 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:29:56.948116 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:30:00.171183 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.171137 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:00.172101 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.172074 4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd59dedc-a03e-408d-8d50-18402ebf9294" containerName="revoke-keys" Nov 01 06:30:00.172202 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.172103 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd59dedc-a03e-408d-8d50-18402ebf9294" containerName="revoke-keys" Nov 01 06:30:00.172390 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.172374 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd59dedc-a03e-408d-8d50-18402ebf9294" containerName="revoke-keys" Nov 01 06:30:00.176752 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.176724 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.198816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.198788 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l"] Nov 01 06:30:00.202547 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.202524 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.229228 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.229197 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:00.247764 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.247723 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:30:00.255716 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.255691 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:30:00.258276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.258235 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.258359 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.258301 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.258410 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.258385 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpb2b\" (UniqueName: \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.258471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.258455 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbb5t\" (UniqueName: \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.258575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.258556 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.280884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.280847 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l"] Nov 01 06:30:00.323265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.323221 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:30:00.323265 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.323221 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:30:00.359827 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.359790 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.360033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.359836 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.360033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.359869 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpb2b\" (UniqueName: \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.360033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.359941 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbb5t\" (UniqueName: \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.360033 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.359983 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.360698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.360618 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.360698 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.360702 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.362652 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.362626 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.422198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.422118 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbb5t\" (UniqueName: \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\") pod \"maas-api-key-cleanup-29891910-z2mhr\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.422198 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.422123 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpb2b\" (UniqueName: \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\") pod \"maas-api-revoke-sub-5cf446cc-mpn9l\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.493035 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.492992 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:00.515596 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.515567 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:00.876041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.876007 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:00.916673 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:30:00.916639 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43a4646_94d2_4268_acbe_a7ffdd448e57.slice/crio-bd6eddc6dfca189517f4017cd88d848b9d8eb3333cd56bf184a12cd3c90f0127 WatchSource:0}: Error finding container bd6eddc6dfca189517f4017cd88d848b9d8eb3333cd56bf184a12cd3c90f0127: Status 404 returned error can't find the container with id bd6eddc6dfca189517f4017cd88d848b9d8eb3333cd56bf184a12cd3c90f0127 Nov 01 06:30:00.926811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:00.926787 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l"] Nov 01 06:30:00.928412 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:30:00.928386 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00dbca0_1787_48d4_a889_479ad2ef0e05.slice/crio-ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc WatchSource:0}: Error finding container ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc: Status 404 returned error can't find the container with id ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc Nov 01 06:30:01.728071 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.728033 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerStarted","Data":"dcdf3fbd9111c336518a7014527b5410ab59d94deaa271ccf3d70127a8b845f3"} Nov 01 06:30:01.728633 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.728100 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerStarted","Data":"bd6eddc6dfca189517f4017cd88d848b9d8eb3333cd56bf184a12cd3c90f0127"} Nov 01 06:30:01.729747 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.729719 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" event={"ID":"b00dbca0-1787-48d4-a889-479ad2ef0e05","Type":"ContainerStarted","Data":"317d228e0e95ff37842f2edc4db19e318c96a55fa6588a486634e76710f3e640"} Nov 01 06:30:01.729876 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.729766 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" event={"ID":"b00dbca0-1787-48d4-a889-479ad2ef0e05","Type":"ContainerStarted","Data":"ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc"} Nov 01 06:30:01.845737 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.845682 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" podStartSLOduration=1.845667014 podStartE2EDuration="1.845667014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.845106926 +0000 UTC m=+5216.985695845" watchObservedRunningTime="2026-11-01 06:30:01.845667014 +0000 UTC m=+5216.986255981" Nov 01 06:30:01.963432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:01.963366 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" podStartSLOduration=1.963353278 podStartE2EDuration="1.963353278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.96291439 +0000 UTC m=+5217.103503308" watchObservedRunningTime="2026-11-01 06:30:01.963353278 +0000 UTC m=+5217.103942197" Nov 01 06:30:02.737824 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:02.737787 4425 generic.go:356] "Generic (PLEG): container finished" podID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerID="dcdf3fbd9111c336518a7014527b5410ab59d94deaa271ccf3d70127a8b845f3" exitCode=60 Nov 01 06:30:02.738365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:02.738037 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerDied","Data":"dcdf3fbd9111c336518a7014527b5410ab59d94deaa271ccf3d70127a8b845f3"} Nov 01 06:30:02.738365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:02.738172 4425 scope.go:122] "RemoveContainer" containerID="dcdf3fbd9111c336518a7014527b5410ab59d94deaa271ccf3d70127a8b845f3" Nov 01 06:30:02.740389 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:02.740358 4425 generic.go:356] "Generic (PLEG): container finished" podID="b00dbca0-1787-48d4-a889-479ad2ef0e05" containerID="317d228e0e95ff37842f2edc4db19e318c96a55fa6588a486634e76710f3e640" exitCode=0 Nov 01 06:30:02.740513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:02.740399 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" event={"ID":"b00dbca0-1787-48d4-a889-479ad2ef0e05","Type":"ContainerDied","Data":"317d228e0e95ff37842f2edc4db19e318c96a55fa6588a486634e76710f3e640"} Nov 01 06:30:03.747324 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.747291 4425 generic.go:356] "Generic (PLEG): container finished" podID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerID="cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1" exitCode=60 Nov 01 06:30:03.747837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.747383 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerDied","Data":"cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1"} Nov 01 06:30:03.747837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.747454 4425 scope.go:122] "RemoveContainer" containerID="dcdf3fbd9111c336518a7014527b5410ab59d94deaa271ccf3d70127a8b845f3" Nov 01 06:30:03.747837 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.747698 4425 scope.go:122] "RemoveContainer" containerID="cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1" Nov 01 06:30:03.748427 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:30:03.748002 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-z2mhr_odh-ai-gateway-infra(c43a4646-94d2-4268-acbe-a7ffdd448e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" Nov 01 06:30:03.888884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.888858 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:03.996829 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.996797 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\") pod \"b00dbca0-1787-48d4-a889-479ad2ef0e05\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " Nov 01 06:30:03.996829 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.996836 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\" (UniqueName: \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\") pod \"b00dbca0-1787-48d4-a889-479ad2ef0e05\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " Nov 01 06:30:03.997068 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.996942 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\") pod \"b00dbca0-1787-48d4-a889-479ad2ef0e05\" (UID: \"b00dbca0-1787-48d4-a889-479ad2ef0e05\") " Nov 01 06:30:03.997392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.997366 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca" pod "b00dbca0-1787-48d4-a889-479ad2ef0e05" (UID: "b00dbca0-1787-48d4-a889-479ad2ef0e05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:03.999264 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.999233 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca" pod "b00dbca0-1787-48d4-a889-479ad2ef0e05" (UID: "b00dbca0-1787-48d4-a889-479ad2ef0e05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:03.999361 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:03.999337 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b" pod "b00dbca0-1787-48d4-a889-479ad2ef0e05" (UID: "b00dbca0-1787-48d4-a889-479ad2ef0e05"). InnerVolumeSpecName "kube-api-access-xpb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:04.097923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.097893 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:04.097923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.097921 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpb2b\" (UniqueName: \"kubernetes.io/projected/b00dbca0-1787-48d4-a889-479ad2ef0e05-kube-api-access-xpb2b\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:04.097923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.097931 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00dbca0-1787-48d4-a889-479ad2ef0e05-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:04.754231 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.754203 4425 scope.go:122] "RemoveContainer" containerID="cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1" Nov 01 06:30:04.754744 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:30:04.754482 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-z2mhr_odh-ai-gateway-infra(c43a4646-94d2-4268-acbe-a7ffdd448e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" Nov 01 06:30:04.755376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.755360 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" Nov 01 06:30:04.755446 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.755386 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l" event={"ID":"b00dbca0-1787-48d4-a889-479ad2ef0e05","Type":"ContainerDied","Data":"ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc"} Nov 01 06:30:04.755446 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:04.755425 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5c39ad1120fc444e0216a7c298ab6fafc57008e9264b9b5f75a2b4a8116fcc" Nov 01 06:30:09.024984 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:09.024950 4425 scope.go:122] "RemoveContainer" containerID="9f90e3f03ecfd7c46f6e5a8891d0982cf8e55bc82158558950dd0870d3433c95" Nov 01 06:30:09.034094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:09.034072 4425 scope.go:122] "RemoveContainer" containerID="82a6225f79650148c4a44b66b483d1d8cd2c9a0de21474aebac3f1f4d6b52fae" Nov 01 06:30:14.969064 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:14.969026 4425 scope.go:122] "RemoveContainer" containerID="cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1" Nov 01 06:30:15.813170 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:15.813136 4425 generic.go:356] "Generic (PLEG): container finished" podID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerID="300b351b225c95de8a5ba78d3b23f25a805e6ac3b6f28337e4b82e8d77dc7f9b" exitCode=60 Nov 01 06:30:15.813378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:15.813210 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerDied","Data":"300b351b225c95de8a5ba78d3b23f25a805e6ac3b6f28337e4b82e8d77dc7f9b"} Nov 01 06:30:15.813378 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:15.813300 4425 scope.go:122] "RemoveContainer" containerID="cbd12b22476528cbc970b18fa7baa38448566733b95087d39b86dc3f5ec129b1" Nov 01 06:30:15.813686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:15.813662 4425 scope.go:122] "RemoveContainer" containerID="300b351b225c95de8a5ba78d3b23f25a805e6ac3b6f28337e4b82e8d77dc7f9b" Nov 01 06:30:15.813937 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:30:15.813917 4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-z2mhr_odh-ai-gateway-infra(c43a4646-94d2-4268-acbe-a7ffdd448e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" Nov 01 06:30:16.940583 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:16.940551 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:17.082180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.082156 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:17.232326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.232218 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\") pod \"c43a4646-94d2-4268-acbe-a7ffdd448e57\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " Nov 01 06:30:17.232509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.232405 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\" (UniqueName: \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\") pod \"c43a4646-94d2-4268-acbe-a7ffdd448e57\" (UID: \"c43a4646-94d2-4268-acbe-a7ffdd448e57\") " Nov 01 06:30:17.232587 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.232563 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca" pod "c43a4646-94d2-4268-acbe-a7ffdd448e57" (UID: "c43a4646-94d2-4268-acbe-a7ffdd448e57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:17.232825 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.232804 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43a4646-94d2-4268-acbe-a7ffdd448e57-maas-api-service-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.234773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.234736 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t" pod "c43a4646-94d2-4268-acbe-a7ffdd448e57" (UID: "c43a4646-94d2-4268-acbe-a7ffdd448e57"). InnerVolumeSpecName "kube-api-access-gbb5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:17.334038 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.333999 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbb5t\" (UniqueName: \"kubernetes.io/projected/c43a4646-94d2-4268-acbe-a7ffdd448e57-kube-api-access-gbb5t\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.826748 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.826713 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" Nov 01 06:30:17.826961 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.826754 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr" event={"ID":"c43a4646-94d2-4268-acbe-a7ffdd448e57","Type":"ContainerDied","Data":"bd6eddc6dfca189517f4017cd88d848b9d8eb3333cd56bf184a12cd3c90f0127"} Nov 01 06:30:17.826961 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.826845 4425 scope.go:122] "RemoveContainer" containerID="300b351b225c95de8a5ba78d3b23f25a805e6ac3b6f28337e4b82e8d77dc7f9b" Nov 01 06:30:17.993544 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:17.993504 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:18.043816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:18.043781 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-z2mhr"] Nov 01 06:30:18.972333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:18.972301 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" path="/var/lib/kubelet/pods/c43a4646-94d2-4268-acbe-a7ffdd448e57/volumes" Nov 01 06:30:26.946303 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:30:26.946245 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:30:35.643329 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643291 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr"] Nov 01 06:30:35.643864 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643803 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.643864 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643822 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.643864 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643854 4425 cpu_manager.go:414] "containerMap: removing container" podUID="b00dbca0-1787-48d4-a889-479ad2ef0e05" containerName="revoke-keys" Nov 01 06:30:35.643864 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643864 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00dbca0-1787-48d4-a889-479ad2ef0e05" containerName="revoke-keys" Nov 01 06:30:35.644015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643886 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.644015 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.643895 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.644082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.644039 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.644082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.644058 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.644082 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.644071 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00dbca0-1787-48d4-a889-479ad2ef0e05" containerName="revoke-keys" Nov 01 06:30:35.644200 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.644082 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:30:35.647204 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.647186 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.745334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.745296 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.745518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.745381 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzqn\" (UniqueName: \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.745518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.745425 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.749092 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.749069 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:30:35.751164 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.751143 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr"] Nov 01 06:30:35.752599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.752570 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:30:35.752599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.752577 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:30:35.752599 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.752594 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:30:35.846083 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.846050 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzqn\" (UniqueName: \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.846100 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.846283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.846166 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.846812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.846789 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.848664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.848645 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.908519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.908435 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzqn\" (UniqueName: \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\") pod \"maas-api-revoke-sub-2a7d6bbe-8wksr\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:35.960735 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:35.960701 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:36.341576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:36.341550 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr"] Nov 01 06:30:36.343359 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:30:36.343326 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32691236_2b26_414e_b442_8e25fde0a23b.slice/crio-32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada WatchSource:0}: Error finding container 32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada: Status 404 returned error can't find the container with id 32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada Nov 01 06:30:36.956988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:36.956900 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" event={"ID":"32691236-2b26-414e-b442-8e25fde0a23b","Type":"ContainerStarted","Data":"52634075b06753ad4212778c8fd81ee682281ecaba1220d7727c4c0b17ee652f"} Nov 01 06:30:36.956988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:36.956965 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" event={"ID":"32691236-2b26-414e-b442-8e25fde0a23b","Type":"ContainerStarted","Data":"32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada"} Nov 01 06:30:37.074983 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:37.074928 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" podStartSLOduration=2.074913758 podStartE2EDuration="2.074913758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:37.07386146 +0000 UTC m=+5252.214450398" watchObservedRunningTime="2026-11-01 06:30:37.074913758 +0000 UTC m=+5252.215502688" Nov 01 06:30:37.963849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:37.963810 4425 generic.go:356] "Generic (PLEG): container finished" podID="32691236-2b26-414e-b442-8e25fde0a23b" containerID="52634075b06753ad4212778c8fd81ee682281ecaba1220d7727c4c0b17ee652f" exitCode=0 Nov 01 06:30:37.964349 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:37.963859 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" event={"ID":"32691236-2b26-414e-b442-8e25fde0a23b","Type":"ContainerDied","Data":"52634075b06753ad4212778c8fd81ee682281ecaba1220d7727c4c0b17ee652f"} Nov 01 06:30:39.111814 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.111782 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:39.178625 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.178597 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\") pod \"32691236-2b26-414e-b442-8e25fde0a23b\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " Nov 01 06:30:39.178809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.178666 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\") pod \"32691236-2b26-414e-b442-8e25fde0a23b\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " Nov 01 06:30:39.178809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.178702 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\" (UniqueName: \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\") pod \"32691236-2b26-414e-b442-8e25fde0a23b\" (UID: \"32691236-2b26-414e-b442-8e25fde0a23b\") " Nov 01 06:30:39.179073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.179048 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca" pod "32691236-2b26-414e-b442-8e25fde0a23b" (UID: "32691236-2b26-414e-b442-8e25fde0a23b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:39.180923 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.180890 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca" pod "32691236-2b26-414e-b442-8e25fde0a23b" (UID: "32691236-2b26-414e-b442-8e25fde0a23b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:39.181074 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.180959 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn" pod "32691236-2b26-414e-b442-8e25fde0a23b" (UID: "32691236-2b26-414e-b442-8e25fde0a23b"). InnerVolumeSpecName "kube-api-access-chzqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:39.279745 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.279658 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:39.279745 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.279689 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32691236-2b26-414e-b442-8e25fde0a23b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:39.279745 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.279700 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzqn\" (UniqueName: \"kubernetes.io/projected/32691236-2b26-414e-b442-8e25fde0a23b-kube-api-access-chzqn\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:39.976062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.976022 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" event={"ID":"32691236-2b26-414e-b442-8e25fde0a23b","Type":"ContainerDied","Data":"32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada"} Nov 01 06:30:39.976062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.976049 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr" Nov 01 06:30:39.976062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:30:39.976074 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f377cb883adf41e2bdab1789799a1f693db05994fa3efb161cb8f631557ada" Nov 01 06:30:57.014900 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:30:57.014623 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:31:07.047330 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:07.047291 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7"] Nov 01 06:31:07.049947 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:07.049918 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456adb91-tk7k7"] Nov 01 06:31:08.971614 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:08.971577 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbecd75b-6fcd-4713-8f51-21272da8b7bf" path="/var/lib/kubelet/pods/cbecd75b-6fcd-4713-8f51-21272da8b7bf/volumes" Nov 01 06:31:09.057917 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:09.057886 4425 scope.go:122] "RemoveContainer" containerID="6e747c6501c58fc52165710cb9e589ad8088762bf9f1ceafa1fc0ce23d61686a" Nov 01 06:31:20.198702 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.198658 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs"] Nov 01 06:31:20.199119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.199058 4425 cpu_manager.go:414] "containerMap: removing container" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:31:20.199119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.199070 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a4646-94d2-4268-acbe-a7ffdd448e57" containerName="cleanup" Nov 01 06:31:20.199119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.199103 4425 cpu_manager.go:414] "containerMap: removing container" podUID="32691236-2b26-414e-b442-8e25fde0a23b" containerName="revoke-keys" Nov 01 06:31:20.199119 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.199108 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32691236-2b26-414e-b442-8e25fde0a23b" containerName="revoke-keys" Nov 01 06:31:20.199279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.199174 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="32691236-2b26-414e-b442-8e25fde0a23b" containerName="revoke-keys" Nov 01 06:31:20.203968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.203947 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.260674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.260645 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs"] Nov 01 06:31:20.294840 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.294805 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jr7\" (UniqueName: \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.295029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.294855 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.295029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.294880 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.308872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.308841 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:31:20.308872 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.308844 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:31:20.309127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.308940 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:31:20.309127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.308995 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:31:20.395763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.395728 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jr7\" (UniqueName: \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.395979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.395789 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.395979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.395826 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.396533 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.396505 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.398419 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.398398 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.457943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.457858 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jr7\" (UniqueName: \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\") pod \"maas-api-revoke-sub-e1d03c91-whpvs\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.516914 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.516877 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:20.891284 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:20.891236 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs"] Nov 01 06:31:20.893298 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:31:20.893270 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3027b8_13d9_4aa1_86ad_51c3362dda88.slice/crio-ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef WatchSource:0}: Error finding container ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef: Status 404 returned error can't find the container with id ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef Nov 01 06:31:21.209366 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:21.209314 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" event={"ID":"1e3027b8-13d9-4aa1-86ad-51c3362dda88","Type":"ContainerStarted","Data":"a88de3a68752f515a3609feeda034858f4b864e86ca6c78f090f651867d984ba"} Nov 01 06:31:21.209861 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:21.209407 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" event={"ID":"1e3027b8-13d9-4aa1-86ad-51c3362dda88","Type":"ContainerStarted","Data":"ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef"} Nov 01 06:31:21.328106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:21.328056 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" podStartSLOduration=1.328043222 podStartE2EDuration="1.328043222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:21.326497444 +0000 UTC m=+5296.467086364" watchObservedRunningTime="2026-11-01 06:31:21.328043222 +0000 UTC m=+5296.468632142" Nov 01 06:31:22.216575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:22.216539 4425 generic.go:356] "Generic (PLEG): container finished" podID="1e3027b8-13d9-4aa1-86ad-51c3362dda88" containerID="a88de3a68752f515a3609feeda034858f4b864e86ca6c78f090f651867d984ba" exitCode=0 Nov 01 06:31:22.217026 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:22.216621 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" event={"ID":"1e3027b8-13d9-4aa1-86ad-51c3362dda88","Type":"ContainerDied","Data":"a88de3a68752f515a3609feeda034858f4b864e86ca6c78f090f651867d984ba"} Nov 01 06:31:23.358387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.358361 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:23.424278 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.424216 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\" (UniqueName: \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\") pod \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " Nov 01 06:31:23.424486 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.424376 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\") pod \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " Nov 01 06:31:23.424486 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.424440 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\") pod \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\" (UID: \"1e3027b8-13d9-4aa1-86ad-51c3362dda88\") " Nov 01 06:31:23.424728 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.424705 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca" pod "1e3027b8-13d9-4aa1-86ad-51c3362dda88" (UID: "1e3027b8-13d9-4aa1-86ad-51c3362dda88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:23.426546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.426517 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7" pod "1e3027b8-13d9-4aa1-86ad-51c3362dda88" (UID: "1e3027b8-13d9-4aa1-86ad-51c3362dda88"). InnerVolumeSpecName "kube-api-access-w6jr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:23.426655 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.426574 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca" pod "1e3027b8-13d9-4aa1-86ad-51c3362dda88" (UID: "1e3027b8-13d9-4aa1-86ad-51c3362dda88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:31:23.526192 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.526098 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6jr7\" (UniqueName: \"kubernetes.io/projected/1e3027b8-13d9-4aa1-86ad-51c3362dda88-kube-api-access-w6jr7\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:23.526192 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.526131 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:23.526192 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:23.526145 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3027b8-13d9-4aa1-86ad-51c3362dda88-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.228293 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:24.228261 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" Nov 01 06:31:24.228471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:24.228289 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs" event={"ID":"1e3027b8-13d9-4aa1-86ad-51c3362dda88","Type":"ContainerDied","Data":"ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef"} Nov 01 06:31:24.228471 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:24.228336 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4bcb9b7a586433f33a85feb910043a531e03352abd850a42c4fe744bf646ef" Nov 01 06:31:26.946189 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:31:26.946146 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:31:30.958230 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:30.958197 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr"] Nov 01 06:31:30.958916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:30.958632 4425 cpu_manager.go:414] "containerMap: removing container" podUID="1e3027b8-13d9-4aa1-86ad-51c3362dda88" containerName="revoke-keys" Nov 01 06:31:30.958916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:30.958647 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3027b8-13d9-4aa1-86ad-51c3362dda88" containerName="revoke-keys" Nov 01 06:31:30.958916 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:30.958749 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3027b8-13d9-4aa1-86ad-51c3362dda88" containerName="revoke-keys" Nov 01 06:31:30.961896 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:30.961875 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.017474 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.017406 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:31:31.068159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.068126 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:31:31.068159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.068127 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:31:31.068159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.068128 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:31:31.070402 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.070376 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr"] Nov 01 06:31:31.101960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.101927 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.102141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.101983 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.102141 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.102082 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nw6\" (UniqueName: \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.203612 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.203580 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.203820 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.203630 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.203820 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.203664 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nw6\" (UniqueName: \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.204460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.204430 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.206135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.206116 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.266326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.266237 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nw6\" (UniqueName: \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\") pod \"maas-api-revoke-sub-95c4c396-xs4fr\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.275117 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.275097 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:31.648797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:31.648776 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr"] Nov 01 06:31:31.650583 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:31:31.650550 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d4c450_e3c7_4368_a6de_785e5c9b2a85.slice/crio-c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986 WatchSource:0}: Error finding container c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986: Status 404 returned error can't find the container with id c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986 Nov 01 06:31:32.272722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:32.272640 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" event={"ID":"50d4c450-e3c7-4368-a6de-785e5c9b2a85","Type":"ContainerStarted","Data":"5cd161b1f5c673176496f8236d0693f2651b2e3b13c095ac489312ca06aca12b"} Nov 01 06:31:32.272722 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:32.272696 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" event={"ID":"50d4c450-e3c7-4368-a6de-785e5c9b2a85","Type":"ContainerStarted","Data":"c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986"} Nov 01 06:31:32.392706 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:32.392647 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" podStartSLOduration=2.392631573 podStartE2EDuration="2.392631573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:32.390463607 +0000 UTC m=+5307.531052556" watchObservedRunningTime="2026-11-01 06:31:32.392631573 +0000 UTC m=+5307.533220492" Nov 01 06:31:33.278816 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:33.278782 4425 generic.go:356] "Generic (PLEG): container finished" podID="50d4c450-e3c7-4368-a6de-785e5c9b2a85" containerID="5cd161b1f5c673176496f8236d0693f2651b2e3b13c095ac489312ca06aca12b" exitCode=0 Nov 01 06:31:33.279237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:33.278864 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" event={"ID":"50d4c450-e3c7-4368-a6de-785e5c9b2a85","Type":"ContainerDied","Data":"5cd161b1f5c673176496f8236d0693f2651b2e3b13c095ac489312ca06aca12b"} Nov 01 06:31:34.418743 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.418718 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:34.537975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.537888 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\") pod \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " Nov 01 06:31:34.538216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.537997 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\") pod \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " Nov 01 06:31:34.538557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.538507 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca" pod "50d4c450-e3c7-4368-a6de-785e5c9b2a85" (UID: "50d4c450-e3c7-4368-a6de-785e5c9b2a85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:34.538809 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.538577 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\" (UniqueName: \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\") pod \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\" (UID: \"50d4c450-e3c7-4368-a6de-785e5c9b2a85\") " Nov 01 06:31:34.538987 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.538959 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:34.540743 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.540719 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca" pod "50d4c450-e3c7-4368-a6de-785e5c9b2a85" (UID: "50d4c450-e3c7-4368-a6de-785e5c9b2a85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:31:34.541660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.541633 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6" pod "50d4c450-e3c7-4368-a6de-785e5c9b2a85" (UID: "50d4c450-e3c7-4368-a6de-785e5c9b2a85"). InnerVolumeSpecName "kube-api-access-n8nw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:34.639611 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.639571 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nw6\" (UniqueName: \"kubernetes.io/projected/50d4c450-e3c7-4368-a6de-785e5c9b2a85-kube-api-access-n8nw6\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:34.639611 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:34.639604 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d4c450-e3c7-4368-a6de-785e5c9b2a85-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:35.290573 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:35.290492 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" Nov 01 06:31:35.290573 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:35.290530 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr" event={"ID":"50d4c450-e3c7-4368-a6de-785e5c9b2a85","Type":"ContainerDied","Data":"c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986"} Nov 01 06:31:35.290767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:31:35.290575 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5ccab10a6502f9798116d882dffbf3be016b132eea8125e1ce625e26fca3986" Nov 01 06:31:56.949796 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:31:56.949750 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:32:05.054127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:05.054091 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t"] Nov 01 06:32:05.104958 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:05.104924 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4hc4t"] Nov 01 06:32:06.971849 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:06.971811 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71cd289-a048-4f7d-a086-cf611204dbe2" path="/var/lib/kubelet/pods/b71cd289-a048-4f7d-a086-cf611204dbe2/volumes" Nov 01 06:32:09.083172 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:09.083139 4425 scope.go:122] "RemoveContainer" containerID="0b1daa6d3aab06e530bcfef40b1d08441df3fb97581c8d2a5cbc5293a84c9fdb" Nov 01 06:32:18.971994 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:18.971923 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc"] Nov 01 06:32:18.972404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:18.972311 4425 cpu_manager.go:414] "containerMap: removing container" podUID="50d4c450-e3c7-4368-a6de-785e5c9b2a85" containerName="revoke-keys" Nov 01 06:32:18.972404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:18.972322 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d4c450-e3c7-4368-a6de-785e5c9b2a85" containerName="revoke-keys" Nov 01 06:32:18.972404 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:18.972392 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d4c450-e3c7-4368-a6de-785e5c9b2a85" containerName="revoke-keys" Nov 01 06:32:18.977276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:18.977239 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.011044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.011017 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.011190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.011057 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5sp\" (UniqueName: \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.011190 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.011148 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.029276 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.029216 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc"] Nov 01 06:32:19.077551 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.077516 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:32:19.077551 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.077535 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:32:19.077551 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.077517 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:32:19.081280 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.081246 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:32:19.112180 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.112147 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.112358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.112190 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5sp\" (UniqueName: \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.112358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.112241 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.112801 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.112776 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.114771 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.114748 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.174832 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.174802 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5sp\" (UniqueName: \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\") pod \"maas-api-revoke-sub-cae5ee1f-zdphc\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.290127 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.290039 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:19.666653 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:19.666627 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc"] Nov 01 06:32:19.668867 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:32:19.668843 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ace48a_2666_4832_8070_eca20437f1f4.slice/crio-5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81 WatchSource:0}: Error finding container 5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81: Status 404 returned error can't find the container with id 5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81 Nov 01 06:32:20.560642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:20.560611 4425 generic.go:356] "Generic (PLEG): container finished" podID="d3ace48a-2666-4832-8070-eca20437f1f4" containerID="923ced8d563e61a4a84652c20e16b8647f85262bbcab243ee3838c23c8eeefae" exitCode=0 Nov 01 06:32:20.561042 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:20.560699 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" event={"ID":"d3ace48a-2666-4832-8070-eca20437f1f4","Type":"ContainerDied","Data":"923ced8d563e61a4a84652c20e16b8647f85262bbcab243ee3838c23c8eeefae"} Nov 01 06:32:20.561042 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:20.560748 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" event={"ID":"d3ace48a-2666-4832-8070-eca20437f1f4","Type":"ContainerStarted","Data":"5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81"} Nov 01 06:32:21.706995 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.706968 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:21.734926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.734895 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\") pod \"d3ace48a-2666-4832-8070-eca20437f1f4\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " Nov 01 06:32:21.735112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.734969 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\" (UniqueName: \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\") pod \"d3ace48a-2666-4832-8070-eca20437f1f4\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " Nov 01 06:32:21.735112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.735036 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\") pod \"d3ace48a-2666-4832-8070-eca20437f1f4\" (UID: \"d3ace48a-2666-4832-8070-eca20437f1f4\") " Nov 01 06:32:21.735366 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.735343 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca" pod "d3ace48a-2666-4832-8070-eca20437f1f4" (UID: "d3ace48a-2666-4832-8070-eca20437f1f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:32:21.735491 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.735475 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:21.737246 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.737227 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca" pod "d3ace48a-2666-4832-8070-eca20437f1f4" (UID: "d3ace48a-2666-4832-8070-eca20437f1f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:32:21.737376 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.737356 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp" pod "d3ace48a-2666-4832-8070-eca20437f1f4" (UID: "d3ace48a-2666-4832-8070-eca20437f1f4"). InnerVolumeSpecName "kube-api-access-9j5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:32:21.836517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.836484 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5sp\" (UniqueName: \"kubernetes.io/projected/d3ace48a-2666-4832-8070-eca20437f1f4-kube-api-access-9j5sp\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:21.836517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:21.836518 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ace48a-2666-4832-8070-eca20437f1f4-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:22.572960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:22.572928 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" Nov 01 06:32:22.573185 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:22.572970 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae5ee1f-zdphc" event={"ID":"d3ace48a-2666-4832-8070-eca20437f1f4","Type":"ContainerDied","Data":"5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81"} Nov 01 06:32:22.573185 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:22.573021 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cdbcd0be7ab7278457aad6e9125bb26ae7840f00aa9014ca4efa45c2e0d2f81" Nov 01 06:32:26.947296 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:32:26.947236 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:32:29.718331 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.718290 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh"] Nov 01 06:32:29.718760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.718731 4425 cpu_manager.go:414] "containerMap: removing container" podUID="d3ace48a-2666-4832-8070-eca20437f1f4" containerName="revoke-keys" Nov 01 06:32:29.718760 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.718744 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ace48a-2666-4832-8070-eca20437f1f4" containerName="revoke-keys" Nov 01 06:32:29.718854 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.718838 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ace48a-2666-4832-8070-eca20437f1f4" containerName="revoke-keys" Nov 01 06:32:29.722122 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.722094 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.778271 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.778176 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:32:29.781775 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.781743 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh"] Nov 01 06:32:29.817686 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.817651 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkddk\" (UniqueName: \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.817910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.817706 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.818021 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.817976 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.828933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.828906 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:32:29.845645 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.845617 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:32:29.845772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.845658 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:32:29.919549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.919506 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.919549 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.919555 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkddk\" (UniqueName: \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.919779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.919589 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.920327 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.920302 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.922077 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.922050 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:29.982287 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:29.982187 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkddk\" (UniqueName: \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\") pod \"maas-api-revoke-sub-de646eb1-djzzh\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:30.042478 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:30.042440 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:30.417851 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:30.417825 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh"] Nov 01 06:32:30.419409 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:32:30.419382 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295bfd8b_40a8_4d46_8159_5f2b04045ca6.slice/crio-76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b WatchSource:0}: Error finding container 76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b: Status 404 returned error can't find the container with id 76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b Nov 01 06:32:30.616485 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:30.616450 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" event={"ID":"295bfd8b-40a8-4d46-8159-5f2b04045ca6","Type":"ContainerStarted","Data":"76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b"} Nov 01 06:32:31.623517 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:31.623480 4425 generic.go:356] "Generic (PLEG): container finished" podID="295bfd8b-40a8-4d46-8159-5f2b04045ca6" containerID="6ac75206c5fc98b2e28e1cb7204c0985d551ce01ed316d3be3059bcc7f4f1d3d" exitCode=0 Nov 01 06:32:31.623960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:31.623565 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" event={"ID":"295bfd8b-40a8-4d46-8159-5f2b04045ca6","Type":"ContainerDied","Data":"6ac75206c5fc98b2e28e1cb7204c0985d551ce01ed316d3be3059bcc7f4f1d3d"} Nov 01 06:32:32.798134 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.798105 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:32.846948 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.846915 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\") pod \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " Nov 01 06:32:32.847123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.847095 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\") pod \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " Nov 01 06:32:32.847196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.847177 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\" (UniqueName: \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\") pod \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\" (UID: \"295bfd8b-40a8-4d46-8159-5f2b04045ca6\") " Nov 01 06:32:32.847417 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.847389 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca" pod "295bfd8b-40a8-4d46-8159-5f2b04045ca6" (UID: "295bfd8b-40a8-4d46-8159-5f2b04045ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:32:32.847613 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.847589 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:32.849504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.849474 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk" pod "295bfd8b-40a8-4d46-8159-5f2b04045ca6" (UID: "295bfd8b-40a8-4d46-8159-5f2b04045ca6"). InnerVolumeSpecName "kube-api-access-dkddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:32:32.849949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.849927 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca" pod "295bfd8b-40a8-4d46-8159-5f2b04045ca6" (UID: "295bfd8b-40a8-4d46-8159-5f2b04045ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:32:32.948296 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.948194 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295bfd8b-40a8-4d46-8159-5f2b04045ca6-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:32.948296 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:32.948226 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkddk\" (UniqueName: \"kubernetes.io/projected/295bfd8b-40a8-4d46-8159-5f2b04045ca6-kube-api-access-dkddk\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:33.636557 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:33.636524 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" Nov 01 06:32:33.636749 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:33.636520 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de646eb1-djzzh" event={"ID":"295bfd8b-40a8-4d46-8159-5f2b04045ca6","Type":"ContainerDied","Data":"76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b"} Nov 01 06:32:33.636749 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:32:33.636656 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d0ace39479f9d589603243836ef1be0593e94d41eea726bca83b757a749a4b" Nov 01 06:32:56.947548 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:32:56.947509 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:33:09.110010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:09.109977 4425 scope.go:122] "RemoveContainer" containerID="817a3ea6fe750b184e731b6644d24eac83bd4b69d87ef5788f5bb1d565413fe9" Nov 01 06:33:20.623531 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.623493 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt"] Nov 01 06:33:20.624075 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.623933 4425 cpu_manager.go:414] "containerMap: removing container" podUID="295bfd8b-40a8-4d46-8159-5f2b04045ca6" containerName="revoke-keys" Nov 01 06:33:20.624075 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.623944 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295bfd8b-40a8-4d46-8159-5f2b04045ca6" containerName="revoke-keys" Nov 01 06:33:20.624075 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.624017 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="295bfd8b-40a8-4d46-8159-5f2b04045ca6" containerName="revoke-keys" Nov 01 06:33:20.628756 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.628735 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.685676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.685639 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.685850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.685730 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6gbg\" (UniqueName: \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.685850 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.685803 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.685934 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.685912 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt"] Nov 01 06:33:20.733009 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.732980 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:33:20.733182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.733014 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:33:20.733182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.733038 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:33:20.733182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.733118 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:33:20.787326 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.787282 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.787544 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.787372 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6gbg\" (UniqueName: \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.787544 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.787431 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.788167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.788092 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.789926 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.789904 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.849699 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.849660 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6gbg\" (UniqueName: \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\") pod \"maas-api-revoke-sub-bad127ea-qqbbt\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:20.942884 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:20.942798 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:21.316381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:21.316350 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt"] Nov 01 06:33:21.317802 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:33:21.317775 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd5f67b_c165_4db0_b0f3_8aebb3fc81a2.slice/crio-a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f WatchSource:0}: Error finding container a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f: Status 404 returned error can't find the container with id a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f Nov 01 06:33:21.933786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:21.933699 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" event={"ID":"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2","Type":"ContainerStarted","Data":"769d9d41873319fc6fd24b9b95b3ecee7143509c0392560b97522ad1189df484"} Nov 01 06:33:21.933786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:21.933754 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" event={"ID":"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2","Type":"ContainerStarted","Data":"a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f"} Nov 01 06:33:22.051140 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:22.051087 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" podStartSLOduration=2.051072763 podStartE2EDuration="2.051072763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:22.050877459 +0000 UTC m=+5417.191466395" watchObservedRunningTime="2026-11-01 06:33:22.051072763 +0000 UTC m=+5417.191661675" Nov 01 06:33:22.940644 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:22.940610 4425 generic.go:356] "Generic (PLEG): container finished" podID="bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" containerID="769d9d41873319fc6fd24b9b95b3ecee7143509c0392560b97522ad1189df484" exitCode=0 Nov 01 06:33:22.941054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:22.940693 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" event={"ID":"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2","Type":"ContainerDied","Data":"769d9d41873319fc6fd24b9b95b3ecee7143509c0392560b97522ad1189df484"} Nov 01 06:33:24.092205 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.092178 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:24.124649 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.124619 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\") pod \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " Nov 01 06:33:24.124772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.124659 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\" (UniqueName: \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\") pod \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " Nov 01 06:33:24.124772 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.124755 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\") pod \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\" (UID: \"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2\") " Nov 01 06:33:24.125036 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.125009 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca" pod "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" (UID: "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:24.126990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.126957 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca" pod "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" (UID: "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:24.127121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.127102 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg" pod "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" (UID: "bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2"). InnerVolumeSpecName "kube-api-access-d6gbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:24.226343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.226223 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:24.226343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.226278 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6gbg\" (UniqueName: \"kubernetes.io/projected/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-kube-api-access-d6gbg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:24.226343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.226294 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:24.959542 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.959510 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" Nov 01 06:33:24.959736 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.959544 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad127ea-qqbbt" event={"ID":"bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2","Type":"ContainerDied","Data":"a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f"} Nov 01 06:33:24.959736 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:24.959619 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a09f77b00a50360ddcaa5064012cd78b1a1abfde1ef005434105c7b9735b005f" Nov 01 06:33:26.947097 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:33:26.947058 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:33:31.629229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.629196 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b"] Nov 01 06:33:31.629949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.629640 4425 cpu_manager.go:414] "containerMap: removing container" podUID="bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" containerName="revoke-keys" Nov 01 06:33:31.629949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.629652 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" containerName="revoke-keys" Nov 01 06:33:31.629949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.629736 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2" containerName="revoke-keys" Nov 01 06:33:31.632939 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.632918 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.690986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.690945 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b"] Nov 01 06:33:31.699147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.699117 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.699337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.699164 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.699337 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.699189 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwdg\" (UniqueName: \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.738641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.738610 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:33:31.738804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.738680 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:33:31.738804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.738695 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:33:31.738804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.738782 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:33:31.800476 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.800442 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.800689 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.800500 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.800689 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.800526 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwdg\" (UniqueName: \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.801610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.801583 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.803455 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.803423 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.863182 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.863150 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwdg\" (UniqueName: \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\") pod \"maas-api-revoke-sub-f884b2ef-tqn6b\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:31.945532 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:31.945444 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:32.325619 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:32.325594 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b"] Nov 01 06:33:32.328043 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:33:32.328010 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f5ba43_bed4_4130_866f_e5318bf58ef9.slice/crio-6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb WatchSource:0}: Error finding container 6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb: Status 404 returned error can't find the container with id 6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb Nov 01 06:33:33.004034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:33.003934 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" event={"ID":"67f5ba43-bed4-4130-866f-e5318bf58ef9","Type":"ContainerStarted","Data":"b7e244de1b43a30fc17f504b57e002b95b73d6ffb7ac0187fdbab14f357092b1"} Nov 01 06:33:33.004034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:33.003989 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" event={"ID":"67f5ba43-bed4-4130-866f-e5318bf58ef9","Type":"ContainerStarted","Data":"6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb"} Nov 01 06:33:33.121232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:33.121178 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" podStartSLOduration=2.121164435 podStartE2EDuration="2.121164435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:33.120870483 +0000 UTC m=+5428.261459423" watchObservedRunningTime="2026-11-01 06:33:33.121164435 +0000 UTC m=+5428.261753351" Nov 01 06:33:34.011045 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:34.011012 4425 generic.go:356] "Generic (PLEG): container finished" podID="67f5ba43-bed4-4130-866f-e5318bf58ef9" containerID="b7e244de1b43a30fc17f504b57e002b95b73d6ffb7ac0187fdbab14f357092b1" exitCode=0 Nov 01 06:33:34.011493 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:34.011103 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" event={"ID":"67f5ba43-bed4-4130-866f-e5318bf58ef9","Type":"ContainerDied","Data":"b7e244de1b43a30fc17f504b57e002b95b73d6ffb7ac0187fdbab14f357092b1"} Nov 01 06:33:35.152921 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.152893 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:35.233703 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.233671 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\") pod \"67f5ba43-bed4-4130-866f-e5318bf58ef9\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " Nov 01 06:33:35.233877 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.233825 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\" (UniqueName: \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\") pod \"67f5ba43-bed4-4130-866f-e5318bf58ef9\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " Nov 01 06:33:35.233877 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.233872 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\") pod \"67f5ba43-bed4-4130-866f-e5318bf58ef9\" (UID: \"67f5ba43-bed4-4130-866f-e5318bf58ef9\") " Nov 01 06:33:35.234323 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.234287 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca" pod "67f5ba43-bed4-4130-866f-e5318bf58ef9" (UID: "67f5ba43-bed4-4130-866f-e5318bf58ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:35.235970 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.235946 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca" pod "67f5ba43-bed4-4130-866f-e5318bf58ef9" (UID: "67f5ba43-bed4-4130-866f-e5318bf58ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:35.236069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.236010 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg" pod "67f5ba43-bed4-4130-866f-e5318bf58ef9" (UID: "67f5ba43-bed4-4130-866f-e5318bf58ef9"). InnerVolumeSpecName "kube-api-access-bdwdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:35.335050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.335018 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwdg\" (UniqueName: \"kubernetes.io/projected/67f5ba43-bed4-4130-866f-e5318bf58ef9-kube-api-access-bdwdg\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:35.335050 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.335047 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:35.335271 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:35.335058 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f5ba43-bed4-4130-866f-e5318bf58ef9-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:36.023968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:36.023921 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" Nov 01 06:33:36.023968 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:36.023957 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f884b2ef-tqn6b" event={"ID":"67f5ba43-bed4-4130-866f-e5318bf58ef9","Type":"ContainerDied","Data":"6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb"} Nov 01 06:33:36.024306 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:36.024008 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f67ca92305e2a9b1ec80836e21e051f97bd4ec30704501e506949fd358b17bb" Nov 01 06:33:42.494821 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.494783 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7"] Nov 01 06:33:42.495213 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.495190 4425 cpu_manager.go:414] "containerMap: removing container" podUID="67f5ba43-bed4-4130-866f-e5318bf58ef9" containerName="revoke-keys" Nov 01 06:33:42.495213 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.495202 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f5ba43-bed4-4130-866f-e5318bf58ef9" containerName="revoke-keys" Nov 01 06:33:42.495315 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.495295 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f5ba43-bed4-4130-866f-e5318bf58ef9" containerName="revoke-keys" Nov 01 06:33:42.498506 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.498484 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.547092 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.547062 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfgc\" (UniqueName: \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.547289 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.547098 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.547289 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.547275 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.553332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.553304 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:33:42.556946 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.556916 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7"] Nov 01 06:33:42.604432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.604398 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:33:42.604432 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.604390 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:33:42.604709 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.604488 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:33:42.647774 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.647740 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.647956 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.647802 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfgc\" (UniqueName: \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.647956 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.647832 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.648673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.648564 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.650333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.650312 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.710352 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.710322 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfgc\" (UniqueName: \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\") pod \"maas-api-revoke-sub-b3136818-lwfk7\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:42.811615 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:42.811592 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:43.185719 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:43.185690 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7"] Nov 01 06:33:43.187211 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:33:43.187173 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693503ae_c658_4455_9ca0_16f92232a7c3.slice/crio-41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2 WatchSource:0}: Error finding container 41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2: Status 404 returned error can't find the container with id 41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2 Nov 01 06:33:44.067779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:44.067740 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" event={"ID":"693503ae-c658-4455-9ca0-16f92232a7c3","Type":"ContainerStarted","Data":"d0a0074f5d0e371345bac1860a1979f37726a4399ac3ed9880a9bda0c05523c7"} Nov 01 06:33:44.068229 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:44.067827 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" event={"ID":"693503ae-c658-4455-9ca0-16f92232a7c3","Type":"ContainerStarted","Data":"41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2"} Nov 01 06:33:44.186967 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:44.186917 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" podStartSLOduration=2.186903368 podStartE2EDuration="2.186903368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:44.185117035 +0000 UTC m=+5439.325705954" watchObservedRunningTime="2026-11-01 06:33:44.186903368 +0000 UTC m=+5439.327492286" Nov 01 06:33:45.074104 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:45.074065 4425 generic.go:356] "Generic (PLEG): container finished" podID="693503ae-c658-4455-9ca0-16f92232a7c3" containerID="d0a0074f5d0e371345bac1860a1979f37726a4399ac3ed9880a9bda0c05523c7" exitCode=0 Nov 01 06:33:45.074523 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:45.074141 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" event={"ID":"693503ae-c658-4455-9ca0-16f92232a7c3","Type":"ContainerDied","Data":"d0a0074f5d0e371345bac1860a1979f37726a4399ac3ed9880a9bda0c05523c7"} Nov 01 06:33:46.217155 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.217131 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:46.280807 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.280769 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\") pod \"693503ae-c658-4455-9ca0-16f92232a7c3\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " Nov 01 06:33:46.280807 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.280807 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\") pod \"693503ae-c658-4455-9ca0-16f92232a7c3\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " Nov 01 06:33:46.281076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.281008 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\" (UniqueName: \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\") pod \"693503ae-c658-4455-9ca0-16f92232a7c3\" (UID: \"693503ae-c658-4455-9ca0-16f92232a7c3\") " Nov 01 06:33:46.281237 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.281206 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca" pod "693503ae-c658-4455-9ca0-16f92232a7c3" (UID: "693503ae-c658-4455-9ca0-16f92232a7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:46.281382 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.281349 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:46.283207 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.283183 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc" pod "693503ae-c658-4455-9ca0-16f92232a7c3" (UID: "693503ae-c658-4455-9ca0-16f92232a7c3"). InnerVolumeSpecName "kube-api-access-pwfgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:46.283480 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.283457 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca" pod "693503ae-c658-4455-9ca0-16f92232a7c3" (UID: "693503ae-c658-4455-9ca0-16f92232a7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:46.382677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.382571 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfgc\" (UniqueName: \"kubernetes.io/projected/693503ae-c658-4455-9ca0-16f92232a7c3-kube-api-access-pwfgc\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:46.382677 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:46.382599 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/693503ae-c658-4455-9ca0-16f92232a7c3-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:47.086291 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:47.086245 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" Nov 01 06:33:47.086291 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:47.086276 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3136818-lwfk7" event={"ID":"693503ae-c658-4455-9ca0-16f92232a7c3","Type":"ContainerDied","Data":"41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2"} Nov 01 06:33:47.086508 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:47.086323 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c4279826e023e52b44235dd636be29e4ecaf8c7441aa43f5fd7d843defb5a2" Nov 01 06:33:53.502959 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.502920 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7"] Nov 01 06:33:53.503433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.503410 4425 cpu_manager.go:414] "containerMap: removing container" podUID="693503ae-c658-4455-9ca0-16f92232a7c3" containerName="revoke-keys" Nov 01 06:33:53.503433 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.503424 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693503ae-c658-4455-9ca0-16f92232a7c3" containerName="revoke-keys" Nov 01 06:33:53.503516 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.503507 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="693503ae-c658-4455-9ca0-16f92232a7c3" containerName="revoke-keys" Nov 01 06:33:53.506705 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.506688 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.557929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.557893 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.557929 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.557933 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tz4\" (UniqueName: \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.558194 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.558090 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.564275 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.564225 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7"] Nov 01 06:33:53.611673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.611638 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:33:53.611673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.611661 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:33:53.611907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.611741 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:33:53.611907 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.611791 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:33:53.659110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.659078 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.659110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.659112 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tz4\" (UniqueName: \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.659334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.659177 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.659794 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.659764 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.661818 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.661791 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.721911 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.721879 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tz4\" (UniqueName: \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\") pod \"maas-api-revoke-sub-bc1069d1-2w9z7\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:53.821449 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:53.821425 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:54.196584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:54.196560 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7"] Nov 01 06:33:54.198491 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:33:54.198456 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1f8cb5_3c07_4efd_a0c0_80377a261be3.slice/crio-3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718 WatchSource:0}: Error finding container 3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718: Status 404 returned error can't find the container with id 3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718 Nov 01 06:33:55.047811 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:55.047782 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf"] Nov 01 06:33:55.098515 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:55.098482 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-hkwgf"] Nov 01 06:33:55.129705 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:55.129669 4425 generic.go:356] "Generic (PLEG): container finished" podID="3c1f8cb5-3c07-4efd-a0c0-80377a261be3" containerID="523a229dacf8f92f3f0458b072299c509d227eff335cf7874c18be940fab82d3" exitCode=0 Nov 01 06:33:55.129879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:55.129744 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" event={"ID":"3c1f8cb5-3c07-4efd-a0c0-80377a261be3","Type":"ContainerDied","Data":"523a229dacf8f92f3f0458b072299c509d227eff335cf7874c18be940fab82d3"} Nov 01 06:33:55.129879 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:55.129792 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" event={"ID":"3c1f8cb5-3c07-4efd-a0c0-80377a261be3","Type":"ContainerStarted","Data":"3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718"} Nov 01 06:33:56.273617 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.273591 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:56.385023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.384940 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\") pod \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " Nov 01 06:33:56.385023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.385007 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\") pod \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " Nov 01 06:33:56.385301 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.385045 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\" (UniqueName: \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\") pod \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\" (UID: \"3c1f8cb5-3c07-4efd-a0c0-80377a261be3\") " Nov 01 06:33:56.385418 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.385394 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca" pod "3c1f8cb5-3c07-4efd-a0c0-80377a261be3" (UID: "3c1f8cb5-3c07-4efd-a0c0-80377a261be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:56.387311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.387286 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca" pod "3c1f8cb5-3c07-4efd-a0c0-80377a261be3" (UID: "3c1f8cb5-3c07-4efd-a0c0-80377a261be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:56.387311 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.387307 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4" pod "3c1f8cb5-3c07-4efd-a0c0-80377a261be3" (UID: "3c1f8cb5-3c07-4efd-a0c0-80377a261be3"). InnerVolumeSpecName "kube-api-access-z2tz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:56.485901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.485862 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:56.485901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.485896 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:56.485901 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.485909 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tz4\" (UniqueName: \"kubernetes.io/projected/3c1f8cb5-3c07-4efd-a0c0-80377a261be3-kube-api-access-z2tz4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:56.947048 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:33:56.947014 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:33:56.973893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:56.973851 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99016e8c-0952-49cd-90a8-fbd7be744f5b" path="/var/lib/kubelet/pods/99016e8c-0952-49cd-90a8-fbd7be744f5b/volumes" Nov 01 06:33:57.141245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:57.141218 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" Nov 01 06:33:57.141451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:57.141248 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1069d1-2w9z7" event={"ID":"3c1f8cb5-3c07-4efd-a0c0-80377a261be3","Type":"ContainerDied","Data":"3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718"} Nov 01 06:33:57.141451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:33:57.141308 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e35d0383ac591bb4b15447444063a15025035c7206f91b14f34652f535d1718" Nov 01 06:34:04.212694 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.212659 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4"] Nov 01 06:34:04.213216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.213113 4425 cpu_manager.go:414] "containerMap: removing container" podUID="3c1f8cb5-3c07-4efd-a0c0-80377a261be3" containerName="revoke-keys" Nov 01 06:34:04.213216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.213127 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1f8cb5-3c07-4efd-a0c0-80377a261be3" containerName="revoke-keys" Nov 01 06:34:04.213216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.213217 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1f8cb5-3c07-4efd-a0c0-80377a261be3" containerName="revoke-keys" Nov 01 06:34:04.216729 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.216704 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.271605 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.271562 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnhw\" (UniqueName: \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.271797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.271673 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.271797 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.271827 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.274842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.274807 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4"] Nov 01 06:34:04.322115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.322070 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:04.322115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.322088 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:04.322115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.322123 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:04.322412 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.322094 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:04.372547 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.372514 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.372734 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.372596 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.372734 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.372673 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnhw\" (UniqueName: \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.373234 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.373209 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.375196 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.375175 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.435556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.435523 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnhw\" (UniqueName: \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\") pod \"maas-api-revoke-sub-079f4ca2-pxbb4\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.533684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.533599 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:04.911389 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:04.911363 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4"] Nov 01 06:34:04.912423 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:04.912391 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05e5cd0_5506_46f6_bd56_03904dcbdc8b.slice/crio-40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de WatchSource:0}: Error finding container 40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de: Status 404 returned error can't find the container with id 40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de Nov 01 06:34:05.196010 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:05.195922 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" event={"ID":"a05e5cd0-5506-46f6-bd56-03904dcbdc8b","Type":"ContainerStarted","Data":"40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de"} Nov 01 06:34:06.204793 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:06.204760 4425 generic.go:356] "Generic (PLEG): container finished" podID="a05e5cd0-5506-46f6-bd56-03904dcbdc8b" containerID="e93d26d3b013d47b4e9b2c3b8e651b1100a9d317f6aad3d47c0b0691662ec69b" exitCode=0 Nov 01 06:34:06.205313 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:06.204852 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" event={"ID":"a05e5cd0-5506-46f6-bd56-03904dcbdc8b","Type":"ContainerDied","Data":"e93d26d3b013d47b4e9b2c3b8e651b1100a9d317f6aad3d47c0b0691662ec69b"} Nov 01 06:34:07.353629 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.353603 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:07.402377 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.402346 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\") pod \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " Nov 01 06:34:07.402575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.402454 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\") pod \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " Nov 01 06:34:07.402575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.402500 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\" (UniqueName: \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\") pod \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\" (UID: \"a05e5cd0-5506-46f6-bd56-03904dcbdc8b\") " Nov 01 06:34:07.402826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.402796 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca" pod "a05e5cd0-5506-46f6-bd56-03904dcbdc8b" (UID: "a05e5cd0-5506-46f6-bd56-03904dcbdc8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:07.404732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.404706 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca" pod "a05e5cd0-5506-46f6-bd56-03904dcbdc8b" (UID: "a05e5cd0-5506-46f6-bd56-03904dcbdc8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:07.404732 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.404719 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw" pod "a05e5cd0-5506-46f6-bd56-03904dcbdc8b" (UID: "a05e5cd0-5506-46f6-bd56-03904dcbdc8b"). InnerVolumeSpecName "kube-api-access-hpnhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:07.503979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.503896 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:07.503979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.503927 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnhw\" (UniqueName: \"kubernetes.io/projected/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-kube-api-access-hpnhw\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:07.503979 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:07.503938 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05e5cd0-5506-46f6-bd56-03904dcbdc8b-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:08.217217 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:08.217184 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" Nov 01 06:34:08.217412 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:08.217216 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4ca2-pxbb4" event={"ID":"a05e5cd0-5506-46f6-bd56-03904dcbdc8b","Type":"ContainerDied","Data":"40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de"} Nov 01 06:34:08.217412 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:08.217298 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40de3df3df509988b5de1bc81d81e4eb2536b94b520fee528fb0fd7913fa68de" Nov 01 06:34:15.217812 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.217770 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w"] Nov 01 06:34:15.218442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.218422 4425 cpu_manager.go:414] "containerMap: removing container" podUID="a05e5cd0-5506-46f6-bd56-03904dcbdc8b" containerName="revoke-keys" Nov 01 06:34:15.218502 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.218448 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05e5cd0-5506-46f6-bd56-03904dcbdc8b" containerName="revoke-keys" Nov 01 06:34:15.218601 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.218590 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05e5cd0-5506-46f6-bd56-03904dcbdc8b" containerName="revoke-keys" Nov 01 06:34:15.222451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.222434 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.278057 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.278023 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w"] Nov 01 06:34:15.280434 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.280406 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.280575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.280453 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7n4\" (UniqueName: \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.280575 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.280528 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.325618 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.325582 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:15.325792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.325670 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:15.326477 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.326456 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:15.326610 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.326505 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:15.381662 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.381631 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.381885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.381676 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7n4\" (UniqueName: \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.381885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.381702 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.382392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.382367 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.384224 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.384205 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.444150 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.444118 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7n4\" (UniqueName: \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\") pod \"maas-api-revoke-sub-86c3d717-v7p6w\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.535653 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.535567 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:15.910642 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:15.910616 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w"] Nov 01 06:34:15.913398 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:15.913372 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f7de88_7b20_4174_b25f_19073ee10bea.slice/crio-7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6 WatchSource:0}: Error finding container 7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6: Status 404 returned error can't find the container with id 7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6 Nov 01 06:34:16.047201 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:16.047155 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq"] Nov 01 06:34:16.050112 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:16.050077 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wb7lq"] Nov 01 06:34:16.260877 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:16.260851 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" event={"ID":"66f7de88-7b20-4174-b25f-19073ee10bea","Type":"ContainerStarted","Data":"7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6"} Nov 01 06:34:16.972415 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:16.972373 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d" path="/var/lib/kubelet/pods/ff96cdb1-8abb-4cf1-9c90-03fbfc4f2f9d/volumes" Nov 01 06:34:17.274574 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:17.274484 4425 generic.go:356] "Generic (PLEG): container finished" podID="66f7de88-7b20-4174-b25f-19073ee10bea" containerID="500bea5a0cb866234914c9742e5edbec7bde798eeec7acf8a52cfb95360fa8f8" exitCode=0 Nov 01 06:34:17.274574 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:17.274563 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" event={"ID":"66f7de88-7b20-4174-b25f-19073ee10bea","Type":"ContainerDied","Data":"500bea5a0cb866234914c9742e5edbec7bde798eeec7acf8a52cfb95360fa8f8"} Nov 01 06:34:18.603292 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.602807 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:18.730955 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.730791 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\" (UniqueName: \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\") pod \"66f7de88-7b20-4174-b25f-19073ee10bea\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " Nov 01 06:34:18.730955 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.730874 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\") pod \"66f7de88-7b20-4174-b25f-19073ee10bea\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " Nov 01 06:34:18.731243 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.731019 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\") pod \"66f7de88-7b20-4174-b25f-19073ee10bea\" (UID: \"66f7de88-7b20-4174-b25f-19073ee10bea\") " Nov 01 06:34:18.731657 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.731624 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca" pod "66f7de88-7b20-4174-b25f-19073ee10bea" (UID: "66f7de88-7b20-4174-b25f-19073ee10bea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:18.734303 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.734239 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca" pod "66f7de88-7b20-4174-b25f-19073ee10bea" (UID: "66f7de88-7b20-4174-b25f-19073ee10bea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:18.735651 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.735621 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4" pod "66f7de88-7b20-4174-b25f-19073ee10bea" (UID: "66f7de88-7b20-4174-b25f-19073ee10bea"). InnerVolumeSpecName "kube-api-access-wt7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:18.832370 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.832330 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:18.832370 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.832370 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f7de88-7b20-4174-b25f-19073ee10bea-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:18.832659 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:18.832386 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7n4\" (UniqueName: \"kubernetes.io/projected/66f7de88-7b20-4174-b25f-19073ee10bea-kube-api-access-wt7n4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:19.312334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:19.305458 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" event={"ID":"66f7de88-7b20-4174-b25f-19073ee10bea","Type":"ContainerDied","Data":"7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6"} Nov 01 06:34:19.312334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:19.305523 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7d96a9279fd70c04842830c5bf690d70e7552c9b464853fa9375b9fe31acc6" Nov 01 06:34:19.312334 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:19.305597 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c3d717-v7p6w" Nov 01 06:34:26.071595 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.071559 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8"] Nov 01 06:34:26.072063 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.072050 4425 cpu_manager.go:414] "containerMap: removing container" podUID="66f7de88-7b20-4174-b25f-19073ee10bea" containerName="revoke-keys" Nov 01 06:34:26.072113 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.072065 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f7de88-7b20-4174-b25f-19073ee10bea" containerName="revoke-keys" Nov 01 06:34:26.072153 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.072142 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f7de88-7b20-4174-b25f-19073ee10bea" containerName="revoke-keys" Nov 01 06:34:26.075285 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.075264 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.180435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.180400 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:26.180435 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.180429 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:26.180674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.180486 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:26.180674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.180509 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:26.183005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.182987 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8"] Nov 01 06:34:26.222156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.222123 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.222313 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.222165 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.222353 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.222335 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmws\" (UniqueName: \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.323484 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.323405 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmws\" (UniqueName: \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.323647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.323516 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.323647 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.323548 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.324245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.324225 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.326034 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.326014 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.385924 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.385895 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmws\" (UniqueName: \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\") pod \"maas-api-revoke-sub-33aa7f73-bwmb8\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.388767 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.388746 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:26.767019 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:26.766991 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8"] Nov 01 06:34:26.767781 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:26.767757 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefe63fa_b4a6_4e01_ae7b_dde28c498b55.slice/crio-a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13 WatchSource:0}: Error finding container a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13: Status 404 returned error can't find the container with id a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13 Nov 01 06:34:26.947984 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:34:26.947950 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:34:27.348631 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:27.348593 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" event={"ID":"aefe63fa-b4a6-4e01-ae7b-dde28c498b55","Type":"ContainerStarted","Data":"11b906a80a6a1136ba95b5af35ba90ddf9432477effe0ee917374b8c1e1e389d"} Nov 01 06:34:27.349029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:27.348668 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" event={"ID":"aefe63fa-b4a6-4e01-ae7b-dde28c498b55","Type":"ContainerStarted","Data":"a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13"} Nov 01 06:34:27.467392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:27.467334 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" podStartSLOduration=1.467317695 podStartE2EDuration="1.467317695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:27.4660103 +0000 UTC m=+5482.606599232" watchObservedRunningTime="2026-11-01 06:34:27.467317695 +0000 UTC m=+5482.607906613" Nov 01 06:34:28.355895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:28.355860 4425 generic.go:356] "Generic (PLEG): container finished" podID="aefe63fa-b4a6-4e01-ae7b-dde28c498b55" containerID="11b906a80a6a1136ba95b5af35ba90ddf9432477effe0ee917374b8c1e1e389d" exitCode=0 Nov 01 06:34:28.355895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:28.355900 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" event={"ID":"aefe63fa-b4a6-4e01-ae7b-dde28c498b55","Type":"ContainerDied","Data":"11b906a80a6a1136ba95b5af35ba90ddf9432477effe0ee917374b8c1e1e389d"} Nov 01 06:34:29.786281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.779842 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:29.820310 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.820234 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\") pod \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " Nov 01 06:34:29.820519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.820382 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\" (UniqueName: \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\") pod \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " Nov 01 06:34:29.820519 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.820502 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\") pod \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\" (UID: \"aefe63fa-b4a6-4e01-ae7b-dde28c498b55\") " Nov 01 06:34:29.821097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.821058 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca" pod "aefe63fa-b4a6-4e01-ae7b-dde28c498b55" (UID: "aefe63fa-b4a6-4e01-ae7b-dde28c498b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:29.823151 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.823122 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws" pod "aefe63fa-b4a6-4e01-ae7b-dde28c498b55" (UID: "aefe63fa-b4a6-4e01-ae7b-dde28c498b55"). InnerVolumeSpecName "kube-api-access-dbmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:29.823392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.823366 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca" pod "aefe63fa-b4a6-4e01-ae7b-dde28c498b55" (UID: "aefe63fa-b4a6-4e01-ae7b-dde28c498b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:29.922288 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.922068 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:29.922288 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.922109 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:29.922288 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:29.922124 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmws\" (UniqueName: \"kubernetes.io/projected/aefe63fa-b4a6-4e01-ae7b-dde28c498b55-kube-api-access-dbmws\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:30.400365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:30.400319 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" event={"ID":"aefe63fa-b4a6-4e01-ae7b-dde28c498b55","Type":"ContainerDied","Data":"a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13"} Nov 01 06:34:30.401283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:30.400649 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75c05b6098997071bf568ee11378874c411e3c69942352eab4fcd9dbd242a13" Nov 01 06:34:30.401283 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:30.400743 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33aa7f73-bwmb8" Nov 01 06:34:37.025189 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.025151 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv"] Nov 01 06:34:37.025796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.025776 4425 cpu_manager.go:414] "containerMap: removing container" podUID="aefe63fa-b4a6-4e01-ae7b-dde28c498b55" containerName="revoke-keys" Nov 01 06:34:37.025851 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.025800 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefe63fa-b4a6-4e01-ae7b-dde28c498b55" containerName="revoke-keys" Nov 01 06:34:37.025941 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.025930 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefe63fa-b4a6-4e01-ae7b-dde28c498b55" containerName="revoke-keys" Nov 01 06:34:37.029740 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.029716 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.083380 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.083349 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:37.084834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.084805 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv"] Nov 01 06:34:37.092835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.092804 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.092975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.092862 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4w6d\" (UniqueName: \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.093054 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.093034 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.132342 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.132311 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:37.132509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.132356 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:37.132509 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.132407 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:37.194211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.194162 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.194211 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.194212 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.194576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.194275 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4w6d\" (UniqueName: \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.195062 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.195032 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.196874 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.196855 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.257126 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.257089 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4w6d\" (UniqueName: \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\") pod \"maas-api-revoke-sub-0379d8c7-6qbbv\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.351935 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.351905 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:37.727106 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:37.727082 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv"] Nov 01 06:34:37.728065 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:37.728035 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d1eb63_9d31_4206_ab5f_a58ceac5db2a.slice/crio-329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3 WatchSource:0}: Error finding container 329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3: Status 404 returned error can't find the container with id 329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3 Nov 01 06:34:38.444044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:38.444008 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" event={"ID":"75d1eb63-9d31-4206-ab5f-a58ceac5db2a","Type":"ContainerStarted","Data":"265bf9a514127383c87c522ff2cd38b8c72fecf1d6405cdd05732efa182a3444"} Nov 01 06:34:38.444468 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:38.444066 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" event={"ID":"75d1eb63-9d31-4206-ab5f-a58ceac5db2a","Type":"ContainerStarted","Data":"329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3"} Nov 01 06:34:38.562076 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:38.562022 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" podStartSLOduration=2.562007466 podStartE2EDuration="2.562007466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:38.561122392 +0000 UTC m=+5493.701711313" watchObservedRunningTime="2026-11-01 06:34:38.562007466 +0000 UTC m=+5493.702596385" Nov 01 06:34:39.453933 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:39.453894 4425 generic.go:356] "Generic (PLEG): container finished" podID="75d1eb63-9d31-4206-ab5f-a58ceac5db2a" containerID="265bf9a514127383c87c522ff2cd38b8c72fecf1d6405cdd05732efa182a3444" exitCode=0 Nov 01 06:34:39.454518 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:39.453956 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" event={"ID":"75d1eb63-9d31-4206-ab5f-a58ceac5db2a","Type":"ContainerDied","Data":"265bf9a514127383c87c522ff2cd38b8c72fecf1d6405cdd05732efa182a3444"} Nov 01 06:34:41.018232 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.017879 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:41.036189 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.036153 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\") pod \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " Nov 01 06:34:41.036425 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.036198 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\" (UniqueName: \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\") pod \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " Nov 01 06:34:41.036425 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.036328 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\") pod \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\" (UID: \"75d1eb63-9d31-4206-ab5f-a58ceac5db2a\") " Nov 01 06:34:41.036673 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.036648 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca" pod "75d1eb63-9d31-4206-ab5f-a58ceac5db2a" (UID: "75d1eb63-9d31-4206-ab5f-a58ceac5db2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:41.038451 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.038430 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca" pod "75d1eb63-9d31-4206-ab5f-a58ceac5db2a" (UID: "75d1eb63-9d31-4206-ab5f-a58ceac5db2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:41.038521 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.038486 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d" pod "75d1eb63-9d31-4206-ab5f-a58ceac5db2a" (UID: "75d1eb63-9d31-4206-ab5f-a58ceac5db2a"). InnerVolumeSpecName "kube-api-access-l4w6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:41.137823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.137782 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:41.137823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.137818 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4w6d\" (UniqueName: \"kubernetes.io/projected/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-kube-api-access-l4w6d\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:41.137823 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.137833 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d1eb63-9d31-4206-ab5f-a58ceac5db2a-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:41.509796 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.509748 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" event={"ID":"75d1eb63-9d31-4206-ab5f-a58ceac5db2a","Type":"ContainerDied","Data":"329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3"} Nov 01 06:34:41.510115 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.510084 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="329d6da0865799b2da9151b27a4666eb81408a9960c73a28d227dd6ce4e948c3" Nov 01 06:34:41.510365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:41.510331 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0379d8c7-6qbbv" Nov 01 06:34:47.786658 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.786617 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x"] Nov 01 06:34:47.787093 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.787079 4425 cpu_manager.go:414] "containerMap: removing container" podUID="75d1eb63-9d31-4206-ab5f-a58ceac5db2a" containerName="revoke-keys" Nov 01 06:34:47.787159 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.787095 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d1eb63-9d31-4206-ab5f-a58ceac5db2a" containerName="revoke-keys" Nov 01 06:34:47.787200 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.787186 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d1eb63-9d31-4206-ab5f-a58ceac5db2a" containerName="revoke-keys" Nov 01 06:34:47.790512 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.790491 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.816275 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.816228 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.816445 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.816400 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.816552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.816536 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbr4\" (UniqueName: \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.844763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.844731 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:47.895442 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.895406 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:47.897938 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.897916 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x"] Nov 01 06:34:47.899640 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.899623 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:47.899741 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.899706 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:47.917340 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.917308 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbr4\" (UniqueName: \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.917498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.917381 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.917498 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.917464 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.918227 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.918203 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.919953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.919921 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:47.979870 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:47.979834 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbr4\" (UniqueName: \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\") pod \"maas-api-revoke-sub-ae03249d-qgk7x\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:48.103832 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:48.103799 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:48.479899 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:48.479862 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x"] Nov 01 06:34:48.481421 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:48.481398 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf379d6e0_0d7b_465a_b35c_b7bba6987c03.slice/crio-a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e WatchSource:0}: Error finding container a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e: Status 404 returned error can't find the container with id a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e Nov 01 06:34:48.548073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:48.548042 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" event={"ID":"f379d6e0-0d7b-465a-b35c-b7bba6987c03","Type":"ContainerStarted","Data":"a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e"} Nov 01 06:34:49.555123 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:49.555087 4425 generic.go:356] "Generic (PLEG): container finished" podID="f379d6e0-0d7b-465a-b35c-b7bba6987c03" containerID="25d493331bd1f5728f17cbdd0ce4bb0c402ea0a2ade27973df9830827ffccce5" exitCode=0 Nov 01 06:34:49.555606 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:49.555182 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" event={"ID":"f379d6e0-0d7b-465a-b35c-b7bba6987c03","Type":"ContainerDied","Data":"25d493331bd1f5728f17cbdd0ce4bb0c402ea0a2ade27973df9830827ffccce5"} Nov 01 06:34:50.734219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.734179 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:50.849858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.849756 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\" (UniqueName: \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\") pod \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " Nov 01 06:34:50.849858 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.849831 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\") pod \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " Nov 01 06:34:50.850133 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.849918 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\") pod \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\" (UID: \"f379d6e0-0d7b-465a-b35c-b7bba6987c03\") " Nov 01 06:34:50.850328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.850294 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca" pod "f379d6e0-0d7b-465a-b35c-b7bba6987c03" (UID: "f379d6e0-0d7b-465a-b35c-b7bba6987c03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:50.854072 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.854041 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4" pod "f379d6e0-0d7b-465a-b35c-b7bba6987c03" (UID: "f379d6e0-0d7b-465a-b35c-b7bba6987c03"). InnerVolumeSpecName "kube-api-access-fpbr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:50.861836 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.857220 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca" pod "f379d6e0-0d7b-465a-b35c-b7bba6987c03" (UID: "f379d6e0-0d7b-465a-b35c-b7bba6987c03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:50.951110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.951069 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:50.951110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.951104 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f379d6e0-0d7b-465a-b35c-b7bba6987c03-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:50.951110 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:50.951118 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbr4\" (UniqueName: \"kubernetes.io/projected/f379d6e0-0d7b-465a-b35c-b7bba6987c03-kube-api-access-fpbr4\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:51.606281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:51.603838 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" event={"ID":"f379d6e0-0d7b-465a-b35c-b7bba6987c03","Type":"ContainerDied","Data":"a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e"} Nov 01 06:34:51.606281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:51.603905 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8804b38e7c0d893f0f6737c1ba2358a7e0a72f2e49277f6befe9bc198f0d93e" Nov 01 06:34:51.606281 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:51.603980 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae03249d-qgk7x" Nov 01 06:34:53.057597 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:53.057559 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z"] Nov 01 06:34:53.108375 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:53.108347 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a601598a-xqx7z"] Nov 01 06:34:54.972420 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:54.972378 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd59dedc-a03e-408d-8d50-18402ebf9294" path="/var/lib/kubelet/pods/dd59dedc-a03e-408d-8d50-18402ebf9294/volumes" Nov 01 06:34:56.946403 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:34:56.946366 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:34:58.535273 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.535226 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj"] Nov 01 06:34:58.535664 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.535650 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f379d6e0-0d7b-465a-b35c-b7bba6987c03" containerName="revoke-keys" Nov 01 06:34:58.535729 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.535665 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f379d6e0-0d7b-465a-b35c-b7bba6987c03" containerName="revoke-keys" Nov 01 06:34:58.535768 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.535734 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f379d6e0-0d7b-465a-b35c-b7bba6987c03" containerName="revoke-keys" Nov 01 06:34:58.539218 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.539201 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.644328 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.644298 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:34:58.644578 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.644463 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:34:58.647045 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.647017 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj"] Nov 01 06:34:58.656282 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.656234 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:34:58.656400 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.656319 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:34:58.668906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.668883 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25b5z\" (UniqueName: \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.669023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.668914 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.669087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.669054 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.770053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.770018 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25b5z\" (UniqueName: \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.770053 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.770052 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.770343 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.770121 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.770781 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.770762 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.772763 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.772732 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.832490 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.832461 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25b5z\" (UniqueName: \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\") pod \"maas-api-revoke-sub-7161992e-rrksj\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:58.852540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:58.852516 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:34:59.228500 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:59.228470 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj"] Nov 01 06:34:59.230108 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:34:59.230080 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc96665_3184_4030_8893_32d0f5a04515.slice/crio-1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1 WatchSource:0}: Error finding container 1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1: Status 404 returned error can't find the container with id 1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1 Nov 01 06:34:59.657537 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:59.657495 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" event={"ID":"9bc96665-3184-4030-8893-32d0f5a04515","Type":"ContainerStarted","Data":"af6c1ab64433fa8e965da683eac002bcd9d665da8c9c2b5bc104d00035b9a416"} Nov 01 06:34:59.657931 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:59.657556 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" event={"ID":"9bc96665-3184-4030-8893-32d0f5a04515","Type":"ContainerStarted","Data":"1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1"} Nov 01 06:34:59.775903 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:34:59.775847 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" podStartSLOduration=1.775830676 podStartE2EDuration="1.775830676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:59.774465787 +0000 UTC m=+5514.915054726" watchObservedRunningTime="2026-11-01 06:34:59.775830676 +0000 UTC m=+5514.916419622" Nov 01 06:35:00.665406 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:00.665370 4425 generic.go:356] "Generic (PLEG): container finished" podID="9bc96665-3184-4030-8893-32d0f5a04515" containerID="af6c1ab64433fa8e965da683eac002bcd9d665da8c9c2b5bc104d00035b9a416" exitCode=0 Nov 01 06:35:00.665831 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:00.665459 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" event={"ID":"9bc96665-3184-4030-8893-32d0f5a04515","Type":"ContainerDied","Data":"af6c1ab64433fa8e965da683eac002bcd9d665da8c9c2b5bc104d00035b9a416"} Nov 01 06:35:01.997710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:01.997628 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:35:02.131026 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.130989 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\") pod \"9bc96665-3184-4030-8893-32d0f5a04515\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " Nov 01 06:35:02.131241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.131034 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\" (UniqueName: \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\") pod \"9bc96665-3184-4030-8893-32d0f5a04515\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " Nov 01 06:35:02.131241 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.131197 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\") pod \"9bc96665-3184-4030-8893-32d0f5a04515\" (UID: \"9bc96665-3184-4030-8893-32d0f5a04515\") " Nov 01 06:35:02.131685 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.131661 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca" pod "9bc96665-3184-4030-8893-32d0f5a04515" (UID: "9bc96665-3184-4030-8893-32d0f5a04515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:02.133641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.133621 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca" pod "9bc96665-3184-4030-8893-32d0f5a04515" (UID: "9bc96665-3184-4030-8893-32d0f5a04515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:02.133786 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.133764 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z" pod "9bc96665-3184-4030-8893-32d0f5a04515" (UID: "9bc96665-3184-4030-8893-32d0f5a04515"). InnerVolumeSpecName "kube-api-access-25b5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:02.232339 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.232299 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:02.232759 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.232339 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25b5z\" (UniqueName: \"kubernetes.io/projected/9bc96665-3184-4030-8893-32d0f5a04515-kube-api-access-25b5z\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:02.232861 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.232770 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc96665-3184-4030-8893-32d0f5a04515-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:02.715286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.715124 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" event={"ID":"9bc96665-3184-4030-8893-32d0f5a04515","Type":"ContainerDied","Data":"1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1"} Nov 01 06:35:02.715286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.715193 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7161992e-rrksj" Nov 01 06:35:02.715286 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.715233 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1533bdbf16f2a8ee0fcfcea1c11f056d03b6fa750e5b703b3ffcbefb4c47f8c1" Nov 01 06:35:02.958513 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.958455 4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-kb7j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:35:02.958842 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:02.958809 4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-kb7j2" podUID="18cb8977-197c-4847-bea0-0d3eea37e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:35:04.048612 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:04.048578 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l"] Nov 01 06:35:04.101233 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:04.101196 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf446cc-mpn9l"] Nov 01 06:35:04.972737 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:04.972708 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00dbca0-1787-48d4-a889-479ad2ef0e05" path="/var/lib/kubelet/pods/b00dbca0-1787-48d4-a889-479ad2ef0e05/volumes" Nov 01 06:35:09.139676 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.139647 4425 scope.go:122] "RemoveContainer" containerID="00ac0faef68f45b22d15da4b530e6531d3c1b3b91e763c063bdd175e36c099b5" Nov 01 06:35:09.281534 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.281501 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c"] Nov 01 06:35:09.281946 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.281933 4425 cpu_manager.go:414] "containerMap: removing container" podUID="9bc96665-3184-4030-8893-32d0f5a04515" containerName="revoke-keys" Nov 01 06:35:09.281993 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.281947 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc96665-3184-4030-8893-32d0f5a04515" containerName="revoke-keys" Nov 01 06:35:09.282029 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.282024 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc96665-3184-4030-8893-32d0f5a04515" containerName="revoke-keys" Nov 01 06:35:09.286370 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.286353 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.390845 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.390748 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:35:09.390845 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.390780 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:35:09.393672 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.393645 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c"] Nov 01 06:35:09.402177 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.402154 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:35:09.402369 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.402181 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:35:09.409007 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.408980 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97b2\" (UniqueName: \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.409135 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.409035 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.409192 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.409174 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.510792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.510756 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.510975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.510807 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.510975 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.510884 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97b2\" (UniqueName: \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.511528 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.511502 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.513365 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.513341 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.573166 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.573134 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97b2\" (UniqueName: \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\") pod \"maas-api-revoke-sub-9bab65e0-sz42c\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.599753 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.599724 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:09.980693 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:09.980665 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c"] Nov 01 06:35:09.981855 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:35:09.981828 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776ce882_c002_45d4_93b7_afb3a7515229.slice/crio-1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a WatchSource:0}: Error finding container 1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a: Status 404 returned error can't find the container with id 1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a Nov 01 06:35:10.762773 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:10.762742 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" event={"ID":"776ce882-c002-45d4-93b7-afb3a7515229","Type":"ContainerStarted","Data":"e8547e0b324036e7e4095c28be7dccbfb88535179d0e692376abe39d79cde30c"} Nov 01 06:35:10.763233 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:10.762799 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" event={"ID":"776ce882-c002-45d4-93b7-afb3a7515229","Type":"ContainerStarted","Data":"1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a"} Nov 01 06:35:10.881073 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:10.881009 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" podStartSLOduration=1.8809953849999999 podStartE2EDuration="1.880995385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:10.880241751 +0000 UTC m=+5526.020830687" watchObservedRunningTime="2026-11-01 06:35:10.880995385 +0000 UTC m=+5526.021584304" Nov 01 06:35:11.772358 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:11.772322 4425 generic.go:356] "Generic (PLEG): container finished" podID="776ce882-c002-45d4-93b7-afb3a7515229" containerID="e8547e0b324036e7e4095c28be7dccbfb88535179d0e692376abe39d79cde30c" exitCode=0 Nov 01 06:35:11.772810 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:11.772376 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" event={"ID":"776ce882-c002-45d4-93b7-afb3a7515229","Type":"ContainerDied","Data":"e8547e0b324036e7e4095c28be7dccbfb88535179d0e692376abe39d79cde30c"} Nov 01 06:35:13.222944 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.222895 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:13.330584 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.330547 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\" (UniqueName: \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\") pod \"776ce882-c002-45d4-93b7-afb3a7515229\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " Nov 01 06:35:13.330792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.330595 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\") pod \"776ce882-c002-45d4-93b7-afb3a7515229\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " Nov 01 06:35:13.330792 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.330614 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\") pod \"776ce882-c002-45d4-93b7-afb3a7515229\" (UID: \"776ce882-c002-45d4-93b7-afb3a7515229\") " Nov 01 06:35:13.330992 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.330968 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca" pod "776ce882-c002-45d4-93b7-afb3a7515229" (UID: "776ce882-c002-45d4-93b7-afb3a7515229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:13.332799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.332773 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca" pod "776ce882-c002-45d4-93b7-afb3a7515229" (UID: "776ce882-c002-45d4-93b7-afb3a7515229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:13.332905 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.332810 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2" pod "776ce882-c002-45d4-93b7-afb3a7515229" (UID: "776ce882-c002-45d4-93b7-afb3a7515229"). InnerVolumeSpecName "kube-api-access-c97b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:13.432001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.431907 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c97b2\" (UniqueName: \"kubernetes.io/projected/776ce882-c002-45d4-93b7-afb3a7515229-kube-api-access-c97b2\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:13.432001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.431939 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:13.432001 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.431949 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776ce882-c002-45d4-93b7-afb3a7515229-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:13.820338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.807806 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" event={"ID":"776ce882-c002-45d4-93b7-afb3a7515229","Type":"ContainerDied","Data":"1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a"} Nov 01 06:35:13.820338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.807867 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbc75a1f5e6acf6449503b85e93d54237074d26db8cf8d26b1673b117b4d98a" Nov 01 06:35:13.820338 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:13.807932 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bab65e0-sz42c" Nov 01 06:35:20.093942 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.093822 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp"] Nov 01 06:35:20.094460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.094295 4425 cpu_manager.go:414] "containerMap: removing container" podUID="776ce882-c002-45d4-93b7-afb3a7515229" containerName="revoke-keys" Nov 01 06:35:20.094460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.094308 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776ce882-c002-45d4-93b7-afb3a7515229" containerName="revoke-keys" Nov 01 06:35:20.094460 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.094381 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="776ce882-c002-45d4-93b7-afb3a7515229" containerName="revoke-keys" Nov 01 06:35:20.097726 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.097705 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.153902 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.153876 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:35:20.157025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.156995 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp"] Nov 01 06:35:20.193949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.193911 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.194121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.193955 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.194216 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.194191 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nll\" (UniqueName: \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.203948 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.203923 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\"" Nov 01 06:35:20.204087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.203956 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:35:20.204087 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.203967 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:35:20.295789 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.295756 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nll\" (UniqueName: \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.295967 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.295827 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.295967 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.295855 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.296626 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.296604 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.298428 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.298412 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.358120 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.358049 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nll\" (UniqueName: \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\") pod \"maas-api-revoke-sub-d74d5a41-j8cjp\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.420138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.420100 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:20.796454 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.796424 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp"] Nov 01 06:35:20.797291 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:35:20.797243 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63a14bf_c218_46da_8913_5e8dd9f993db.slice/crio-e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef WatchSource:0}: Error finding container e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef: Status 404 returned error can't find the container with id e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef Nov 01 06:35:20.844909 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:20.844876 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" event={"ID":"f63a14bf-c218-46da-8913-5e8dd9f993db","Type":"ContainerStarted","Data":"e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef"} Nov 01 06:35:21.852556 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:21.852522 4425 generic.go:356] "Generic (PLEG): container finished" podID="f63a14bf-c218-46da-8913-5e8dd9f993db" containerID="38148ae8f33440eee43c59e0d483cd4cd60fc28e3032166298ef9eb372792579" exitCode=0 Nov 01 06:35:21.852949 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:21.852609 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" event={"ID":"f63a14bf-c218-46da-8913-5e8dd9f993db","Type":"ContainerDied","Data":"38148ae8f33440eee43c59e0d483cd4cd60fc28e3032166298ef9eb372792579"} Nov 01 06:35:23.294316 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.284819 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:23.328782 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.328754 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\") pod \"f63a14bf-c218-46da-8913-5e8dd9f993db\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " Nov 01 06:35:23.328953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.328799 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\") pod \"f63a14bf-c218-46da-8913-5e8dd9f993db\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " Nov 01 06:35:23.328953 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.328920 4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\" (UniqueName: \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\") pod \"f63a14bf-c218-46da-8913-5e8dd9f993db\" (UID: \"f63a14bf-c218-46da-8913-5e8dd9f993db\") " Nov 01 06:35:23.329333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.329302 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca" pod "f63a14bf-c218-46da-8913-5e8dd9f993db" (UID: "f63a14bf-c218-46da-8913-5e8dd9f993db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:23.331152 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.331121 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca" pod "f63a14bf-c218-46da-8913-5e8dd9f993db" (UID: "f63a14bf-c218-46da-8913-5e8dd9f993db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:23.331309 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.331289 4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll" pod "f63a14bf-c218-46da-8913-5e8dd9f993db" (UID: "f63a14bf-c218-46da-8913-5e8dd9f993db"). InnerVolumeSpecName "kube-api-access-p8nll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:23.430023 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.429985 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-xks-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:23.430199 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.430027 4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f63a14bf-c218-46da-8913-5e8dd9f993db-maas-api-openshift-ca\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:23.430199 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.430049 4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nll\" (UniqueName: \"kubernetes.io/projected/f63a14bf-c218-46da-8913-5e8dd9f993db-kube-api-access-p8nll\") on node \"ip-10-0-141-97.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:23.897538 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.897497 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" event={"ID":"f63a14bf-c218-46da-8913-5e8dd9f993db","Type":"ContainerDied","Data":"e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef"} Nov 01 06:35:23.897851 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.897828 4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e36c386cf0d5e16e28f2ad3d06788915cffc47aafb234f330817d922717e52ef" Nov 01 06:35:23.898044 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:23.898028 4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d5a41-j8cjp" Nov 01 06:35:26.948099 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:35:26.947989 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:35:40.048674 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:40.048637 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr"] Nov 01 06:35:40.099893 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:40.099862 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7d6bbe-8wksr"] Nov 01 06:35:40.972799 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:40.972763 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32691236-2b26-414e-b442-8e25fde0a23b" path="/var/lib/kubelet/pods/32691236-2b26-414e-b442-8e25fde0a23b/volumes" Nov 01 06:35:51.776084 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:35:51.776047 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-847cf6dc9d-lk8pc_085b04bc-51a6-4928-815d-f38fe27aea3d/authorino/0.log" Nov 01 06:35:56.947741 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:35:56.947700 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:36:03.302462 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:03.302425 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/1.log" Nov 01 06:36:06.464438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:06.464409 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/0.log" Nov 01 06:36:07.770660 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:07.770633 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0379d8c7-6qbbv_75d1eb63-9d31-4206-ab5f-a58ceac5db2a/revoke-keys/0.log" Nov 01 06:36:07.974070 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:07.974045 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-079f4ca2-pxbb4_a05e5cd0-5506-46f6-bd56-03904dcbdc8b/revoke-keys/0.log" Nov 01 06:36:08.194069 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:08.194040 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-33aa7f73-bwmb8_aefe63fa-b4a6-4e01-ae7b-dde28c498b55/revoke-keys/0.log" Nov 01 06:36:08.405504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:08.405475 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7161992e-rrksj_9bc96665-3184-4030-8893-32d0f5a04515/revoke-keys/0.log" Nov 01 06:36:08.635988 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:08.635962 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-86c3d717-v7p6w_66f7de88-7b20-4174-b25f-19073ee10bea/revoke-keys/0.log" Nov 01 06:36:08.860185 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:08.860156 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-95c4c396-xs4fr_50d4c450-e3c7-4368-a6de-785e5c9b2a85/revoke-keys/0.log" Nov 01 06:36:09.066913 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.066887 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9bab65e0-sz42c_776ce882-c002-45d4-93b7-afb3a7515229/revoke-keys/0.log" Nov 01 06:36:09.161986 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.161953 4425 scope.go:122] "RemoveContainer" containerID="7410e1d554413ce00978bf31617f71ef546f4574984df90e51719e9d8cd9e3a8" Nov 01 06:36:09.171142 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.171119 4425 scope.go:122] "RemoveContainer" containerID="317d228e0e95ff37842f2edc4db19e318c96a55fa6588a486634e76710f3e640" Nov 01 06:36:09.179443 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.179424 4425 scope.go:122] "RemoveContainer" containerID="5e835ebeff36f4dc9443eb9ae207f9c1adea51ac2b7d745b58f2b8d45b94268e" Nov 01 06:36:09.286564 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.286536 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ae03249d-qgk7x_f379d6e0-0d7b-465a-b35c-b7bba6987c03/revoke-keys/0.log" Nov 01 06:36:09.500608 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.500531 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3136818-lwfk7_693503ae-c658-4455-9ca0-16f92232a7c3/revoke-keys/0.log" Nov 01 06:36:09.708264 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.708236 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bad127ea-qqbbt_bbd5f67b-c165-4db0-b0f3-8aebb3fc81a2/revoke-keys/0.log" Nov 01 06:36:09.921814 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:09.921785 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bc1069d1-2w9z7_3c1f8cb5-3c07-4efd-a0c0-80377a261be3/revoke-keys/0.log" Nov 01 06:36:10.140672 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:10.140643 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cae5ee1f-zdphc_d3ace48a-2666-4832-8070-eca20437f1f4/revoke-keys/0.log" Nov 01 06:36:10.354641 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:10.354613 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d74d5a41-j8cjp_f63a14bf-c218-46da-8913-5e8dd9f993db/revoke-keys/0.log" Nov 01 06:36:10.570977 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:10.570951 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-de646eb1-djzzh_295bfd8b-40a8-4d46-8159-5f2b04045ca6/revoke-keys/0.log" Nov 01 06:36:10.790167 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:10.790081 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e1d03c91-whpvs_1e3027b8-13d9-4aa1-86ad-51c3362dda88/revoke-keys/0.log" Nov 01 06:36:11.008357 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:11.008328 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f884b2ef-tqn6b_67f5ba43-bed4-4130-866f-e5318bf58ef9/revoke-keys/0.log" Nov 01 06:36:13.218696 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:13.218659 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-847cf6dc9d-lk8pc_085b04bc-51a6-4928-815d-f38fe27aea3d/authorino/0.log" Nov 01 06:36:16.774504 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:16.774472 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f68b7bd84-nfg9z_dd2e9d9d-ffe8-44c0-8a9f-726020910c14/kube-auth-proxy/0.log" Nov 01 06:36:17.004394 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:17.004357 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-8ng9q_642bdc4b-db0b-4f67-aab4-aa429c34b921/istio-proxy/0.log" Nov 01 06:36:17.272025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:17.271994 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-b4bkr_61a296e9-2e08-439f-96cc-e2c46a96a22c/istio-proxy/0.log" Nov 01 06:36:18.438333 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:18.438302 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-kb7j2_18cb8977-197c-4847-bea0-0d3eea37e625/istio-proxy/0.log" Nov 01 06:36:20.066605 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:20.066567 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-55775b4dc9-fvzqg_fbb1ee25-949c-4971-aae8-944e3618f16d/payload-pre-processing/0.log" Nov 01 06:36:20.271381 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:20.271344 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b6ff555c6-lxd69_c798be11-e998-4882-9ce6-6806a72664a2/payload-processing/0.log" Nov 01 06:36:22.052906 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:22.052872 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mf4nj_fc8a7b4f-056c-4649-9b91-f4f625e9e24c/main/0.log" Nov 01 06:36:22.769895 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:22.769863 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mgng5_18a5d029-b30e-40a5-ba11-40e5d13560bc/main/0.log" Nov 01 06:36:23.261608 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:23.261578 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5rztr_61c1575d-d0b3-443d-8b98-812b59888456/main/0.log" Nov 01 06:36:23.521443 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:23.521357 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xj4gz_27bad948-9ce6-42ff-b5e2-3deaa8d02069/main/0.log" Nov 01 06:36:24.059147 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:24.059112 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs"] Nov 01 06:36:24.110161 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:24.110123 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d03c91-whpvs"] Nov 01 06:36:24.972319 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:24.972290 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3027b8-13d9-4aa1-86ad-51c3362dda88" path="/var/lib/kubelet/pods/1e3027b8-13d9-4aa1-86ad-51c3362dda88/volumes" Nov 01 06:36:26.946969 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:36:26.946930 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:36:33.875616 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.875582 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l6mr9/must-gather-q9dj6"] Nov 01 06:36:33.876192 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.876172 4425 cpu_manager.go:414] "containerMap: removing container" podUID="f63a14bf-c218-46da-8913-5e8dd9f993db" containerName="revoke-keys" Nov 01 06:36:33.876313 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.876195 4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63a14bf-c218-46da-8913-5e8dd9f993db" containerName="revoke-keys" Nov 01 06:36:33.876387 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.876370 4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63a14bf-c218-46da-8913-5e8dd9f993db" containerName="revoke-keys" Nov 01 06:36:33.880156 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.880132 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:33.935719 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.935687 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l6mr9/must-gather-q9dj6"] Nov 01 06:36:33.984025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.983990 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l6mr9\"/\"openshift-service-ca.crt\"" Nov 01 06:36:33.984025 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.984002 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l6mr9\"/\"kube-root-ca.crt\"" Nov 01 06:36:33.984279 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:33.984080 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l6mr9\"/\"default-dockercfg-v64hp\"" Nov 01 06:36:34.072046 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.072015 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8st8\" (UniqueName: \"kubernetes.io/projected/428cd950-e9b7-4ed9-b62b-1f86b9404779-kube-api-access-v8st8\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.072248 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.072104 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/428cd950-e9b7-4ed9-b62b-1f86b9404779-must-gather-output\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.173540 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.173454 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/428cd950-e9b7-4ed9-b62b-1f86b9404779-must-gather-output\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.173710 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.173560 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8st8\" (UniqueName: \"kubernetes.io/projected/428cd950-e9b7-4ed9-b62b-1f86b9404779-kube-api-access-v8st8\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.173845 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.173828 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/428cd950-e9b7-4ed9-b62b-1f86b9404779-must-gather-output\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.235835 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.235801 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8st8\" (UniqueName: \"kubernetes.io/projected/428cd950-e9b7-4ed9-b62b-1f86b9404779-kube-api-access-v8st8\") pod \"must-gather-q9dj6\" (UID: \"428cd950-e9b7-4ed9-b62b-1f86b9404779\") " pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.494779 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.494684 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" Nov 01 06:36:34.875491 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.875466 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l6mr9/must-gather-q9dj6"] Nov 01 06:36:34.876662 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:36:34.876631 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod428cd950_e9b7_4ed9_b62b_1f86b9404779.slice/crio-d6ad7154842e4cc8697a485bbb05f4b5219a2606b1e12517089715920fcab83c WatchSource:0}: Error finding container d6ad7154842e4cc8697a485bbb05f4b5219a2606b1e12517089715920fcab83c: Status 404 returned error can't find the container with id d6ad7154842e4cc8697a485bbb05f4b5219a2606b1e12517089715920fcab83c Nov 01 06:36:34.879138 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:34.879119 4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:36:35.051608 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:35.051572 4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr"] Nov 01 06:36:35.053742 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:35.053721 4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c4c396-xs4fr"] Nov 01 06:36:35.341136 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:35.341103 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" event={"ID":"428cd950-e9b7-4ed9-b62b-1f86b9404779","Type":"ContainerStarted","Data":"d6ad7154842e4cc8697a485bbb05f4b5219a2606b1e12517089715920fcab83c"} Nov 01 06:36:36.973219 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:36.973176 4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d4c450-e3c7-4368-a6de-785e5c9b2a85" path="/var/lib/kubelet/pods/50d4c450-e3c7-4368-a6de-785e5c9b2a85/volumes" Nov 01 06:36:42.392008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:42.391963 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" event={"ID":"428cd950-e9b7-4ed9-b62b-1f86b9404779","Type":"ContainerStarted","Data":"96832772c597a3064149719e5c11404efb13a4bb735e7ebfdfd642c30866d127"} Nov 01 06:36:42.392008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:42.392010 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" event={"ID":"428cd950-e9b7-4ed9-b62b-1f86b9404779","Type":"ContainerStarted","Data":"f2a93ebc13532765f73977af63d6c03ec7f59b393766154a2fba7b03c41442df"} Nov 01 06:36:42.510052 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:42.509994 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l6mr9/must-gather-q9dj6" podStartSLOduration=2.362148858 podStartE2EDuration="9.509978788s" totalImagesPullingTime="7.14782993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:42.508695721 +0000 UTC m=+5617.649284640" watchObservedRunningTime="2026-11-01 06:36:42.509978788 +0000 UTC m=+5617.650567700" Nov 01 06:36:43.426798 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:43.426764 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kwpzs_d20a9070-33a8-4606-b4c4-3c38e2b4abe0/global-pull-secret-syncer/0.log" Nov 01 06:36:46.284639 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:46.284605 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/1.log" Nov 01 06:36:46.313065 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:46.313034 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/0.log" Nov 01 06:36:47.017936 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:47.017907 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mhnv9_d89f9f69-4ddd-4676-a02b-d836a0cf627e/konnectivity-agent/0.log" Nov 01 06:36:48.266157 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:48.266116 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-97.us-west-1.compute.internal_0733015cf0f4fb0a3c7bd42111c0f986/haproxy/0.log" Nov 01 06:36:51.949041 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:51.949003 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/1.log" Nov 01 06:36:52.002834 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:52.002805 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/0.log" Nov 01 06:36:53.066175 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.066136 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-847cf6dc9d-lk8pc_085b04bc-51a6-4928-815d-f38fe27aea3d/authorino/0.log" Nov 01 06:36:53.183804 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.183767 4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs"] Nov 01 06:36:53.187503 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.187486 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.268374 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.268342 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-proc\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.268552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.268412 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-lib-modules\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.268552 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.268498 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-podres\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.268644 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.268616 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-sys\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.268695 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.268639 4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9txt\" (UniqueName: \"kubernetes.io/projected/74a8da1d-f668-4629-af0d-790b23d41c84-kube-api-access-d9txt\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.278438 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.278408 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs"] Nov 01 06:36:53.330005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.329928 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9nzz\"/\"kube-root-ca.crt\"" Nov 01 06:36:53.330005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.329963 4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b9nzz\"/\"default-dockercfg-87f45\"" Nov 01 06:36:53.330005 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.329963 4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9nzz\"/\"openshift-service-ca.crt\"" Nov 01 06:36:53.369488 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369454 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-podres\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369540 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-sys\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369563 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9txt\" (UniqueName: \"kubernetes.io/projected/74a8da1d-f668-4629-af0d-790b23d41c84-kube-api-access-d9txt\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369585 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-podres\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369656 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369598 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-proc\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369662 4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-lib-modules\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369675 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-sys\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369675 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-proc\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.369826 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.369758 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74a8da1d-f668-4629-af0d-790b23d41c84-lib-modules\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.434008 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.433969 4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9txt\" (UniqueName: \"kubernetes.io/projected/74a8da1d-f668-4629-af0d-790b23d41c84-kube-api-access-d9txt\") pod \"perf-node-gather-daemonset-6zgrs\" (UID: \"74a8da1d-f668-4629-af0d-790b23d41c84\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.499718 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.499685 4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:53.881391 ip-10-0-141-97 kubenswrapper[4425]: W1101 06:36:53.881351 4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74a8da1d_f668_4629_af0d_790b23d41c84.slice/crio-15c9f33da75302f63e2765600edaf6e117ee653ca29337ee8a04d7196a63e744 WatchSource:0}: Error finding container 15c9f33da75302f63e2765600edaf6e117ee653ca29337ee8a04d7196a63e744: Status 404 returned error can't find the container with id 15c9f33da75302f63e2765600edaf6e117ee653ca29337ee8a04d7196a63e744 Nov 01 06:36:53.881592 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:53.881572 4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs"] Nov 01 06:36:54.463777 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:54.463736 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" event={"ID":"74a8da1d-f668-4629-af0d-790b23d41c84","Type":"ContainerStarted","Data":"acb8a73a18b2d512afe4fb739e4ca2cf866a05dd0c65d7ea21c756a7bd317a0a"} Nov 01 06:36:54.464274 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:54.463816 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:36:54.464274 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:54.463842 4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" event={"ID":"74a8da1d-f668-4629-af0d-790b23d41c84","Type":"ContainerStarted","Data":"15c9f33da75302f63e2765600edaf6e117ee653ca29337ee8a04d7196a63e744"} Nov 01 06:36:54.583607 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:54.583548 4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" podStartSLOduration=1.5835334479999998 podStartE2EDuration="1.583533448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:54.581905198 +0000 UTC m=+5629.722494117" watchObservedRunningTime="2026-11-01 06:36:54.583533448 +0000 UTC m=+5629.724122370" Nov 01 06:36:56.947546 ip-10-0-141-97 kubenswrapper[4425]: E1101 06:36:56.947508 4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2r68" containerName="extractor" Nov 01 06:36:58.247423 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:58.247388 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/1.log" Nov 01 06:36:58.276335 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:58.276308 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-559cb95fbf-czdwv_9e4d7b1c-a9bf-439c-b355-4149893982d5/manager/0.log" Nov 01 06:36:59.002885 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:59.002813 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/alertmanager/0.log" Nov 01 06:36:59.417392 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:59.417366 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/config-reloader/0.log" Nov 01 06:36:59.831734 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:36:59.831700 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/kube-rbac-proxy-web/0.log" Nov 01 06:37:00.246684 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:00.246603 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/kube-rbac-proxy/0.log" Nov 01 06:37:00.482829 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:00.482597 4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-6zgrs" Nov 01 06:37:00.664576 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:00.664547 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/kube-rbac-proxy-metric/0.log" Nov 01 06:37:01.079103 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:01.079078 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/prom-label-proxy/0.log" Nov 01 06:37:01.493121 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:01.493036 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5aeb4ea-7f78-4711-be7b-e0b736015b9a/init-config-reloader/0.log" Nov 01 06:37:04.003097 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:04.003067 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2vbms_2278e4b7-d92f-491a-b98b-ad1203cd8b9c/monitoring-plugin/0.log" Nov 01 06:37:05.883245 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:05.883208 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f68b7bd84-nfg9z_dd2e9d9d-ffe8-44c0-8a9f-726020910c14/kube-auth-proxy/0.log" Nov 01 06:37:06.103321 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:06.103288 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-8ng9q_642bdc4b-db0b-4f67-aab4-aa429c34b921/istio-proxy/0.log" Nov 01 06:37:06.321726 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:06.321685 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-b4bkr_61a296e9-2e08-439f-96cc-e2c46a96a22c/istio-proxy/0.log" Nov 01 06:37:06.924332 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:06.924302 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsvch_048500be-6716-42e5-93e3-e3b7e3507882/node-exporter/0.log" Nov 01 06:37:07.096960 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:07.096923 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-kb7j2_18cb8977-197c-4847-bea0-0d3eea37e625/istio-proxy/0.log" Nov 01 06:37:07.344938 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:07.344911 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsvch_048500be-6716-42e5-93e3-e3b7e3507882/kube-rbac-proxy/0.log" Nov 01 06:37:07.758910 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:07.758842 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsvch_048500be-6716-42e5-93e3-e3b7e3507882/init-textfile/0.log" Nov 01 06:37:08.175081 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:08.175051 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rf9s2_d5ddfea2-3759-4ad2-9b50-48acaf309f1f/kube-rbac-proxy-main/0.log" Nov 01 06:37:08.376943 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:08.376909 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-55775b4dc9-fvzqg_fbb1ee25-949c-4971-aae8-944e3618f16d/payload-pre-processing/0.log" Nov 01 06:37:08.586620 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:08.586592 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rf9s2_d5ddfea2-3759-4ad2-9b50-48acaf309f1f/kube-rbac-proxy-self/0.log" Nov 01 06:37:08.588354 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:08.588330 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b6ff555c6-lxd69_c798be11-e998-4882-9ce6-6806a72664a2/payload-processing/0.log" Nov 01 06:37:09.011546 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:09.011460 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rf9s2_d5ddfea2-3759-4ad2-9b50-48acaf309f1f/openshift-state-metrics/0.log" Nov 01 06:37:09.209109 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:09.209070 4425 scope.go:122] "RemoveContainer" containerID="52634075b06753ad4212778c8fd81ee682281ecaba1220d7727c4c0b17ee652f" Nov 01 06:37:09.443639 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:09.443611 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/prometheus/0.log" Nov 01 06:37:09.855094 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:09.855069 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/config-reloader/0.log" Nov 01 06:37:10.268088 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:10.268004 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/thanos-sidecar/0.log" Nov 01 06:37:10.683461 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:10.683428 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/kube-rbac-proxy-web/0.log" Nov 01 06:37:11.095990 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:11.095963 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/kube-rbac-proxy/0.log" Nov 01 06:37:11.511719 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:11.511635 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/kube-rbac-proxy-thanos/0.log" Nov 01 06:37:11.925589 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:11.925552 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92f00763-56d4-448a-92c8-278e091e1206/init-config-reloader/0.log" Nov 01 06:37:12.352244 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:12.352209 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-trvcp_9a68393e-64c6-46d6-9cb5-03a2d1324c60/prometheus-operator/0.log" Nov 01 06:37:12.757118 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:12.757034 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-trvcp_9a68393e-64c6-46d6-9cb5-03a2d1324c60/kube-rbac-proxy/0.log" Nov 01 06:37:14.820855 ip-10-0-141-97 kubenswrapper[4425]: I1101 06:37:14.820817 4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-847cf6dc9d-lk8pc_085b04bc-51a6-4928-815d-f38fe27aea3d/authorino/0.log"